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ABSTRACT 

The first objective of a tvo-phase project was to 
clarify the policy rationale and intervention model for 
state-fina'.ced vorKplace-hased retraining programs in terms of their 
two objectives of retaining jobs and preventing unemployment. The 
second objective was to develops (1) an evaluation system for these 
programs that was composed of business screening guidelines for 
targeting investment and minimizing substitution effects; and (2) a 
perforsrance outcome system for measuring program outcomes that are 
necessary to achieve these two program objectives. In pl»se one, the 
project reviewed California's Employment Training Panel and Illinois' 
Prairie state 2000 Authority and determined that a successful 
training project should result in these perfonmince outcomes: 
training objectives* work unit performance, cranpany performance, and 
trainee earnings. Based on this intervention model, the first phase 
of the project also developed business screening guidelines that 
would maximize the effects of training projects on program objectives 
and minimize substitution problems. The second phase of the study 
evaluated the feasibility of implementing the evaluation model in 
four state-financed, worXplace-based retraining programs and examined 
24 case studies from California* Illinois, Hissouri, and New York. 
The project showed the feasibility of a state evaluation system and 
made recommendations for a 3-year implementation plan. (Appendices 
include the following! (1) descriptions of the four state programs; 
(2) sampling design and methodology; (3) impact evaluation of 
retraining programs; and (4) 15 references.) (KC) 
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Preface 



This Import is divided into two volumes. Volume I describes the proposwi 
business screening and performance outccMiie evaluation system, our find- 
ings regarding the feasibiUty of the system, and our recommendations for 
inwlementation. Volume D is a compendium of 24 case studies that were done 
totestthefeasibiUtyofthescreeningandevaluationsystem. They also illustrate 
the variety of retraining investments made by state governments and provide 
practical lessons for deigning successful retraining efforts. 
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EXECUTIVE SUMMARY 



Since the t960s, states have created 
customized training programs as part of 
their economic development efforts to 
attract, expand, and retain businesses. 
Unlike traditional employment and 
training programs that focus on 
individuals and specific target 
populations, sUte-financed customized 
tiaifting programs are mainly economic 
development programs- They emphasize 
business training needs and the use of state 
training funds in increasing and retaining 
jobs for the state. Most recent studies 
estimate that 46 states now have one or 
more customized training programs. 

State-financed, customized training 
programs emerged in the 1960s and 1970s 
as part of state efforts to attract new 
businesses to the state and promote 
employment expansion in existing 
business^. These state programs focused 
on training unemployed or newly hired 
workers for new jobs. State-financed, 
workplace-based retraining programs, 
that is, programs that focus on business 
retention and the retraining of existing 
employees are a phenomenon of the 1980s. 
During the bst decade, buslnes.* retention 
and employee retraining have become 
significant components of most 
customized training programs. 

State-financed, workplace-based 
retraining programs have been established 
on the assumption that the most effective 
way to address unemployment problems 
in states Is to prevent them in the P»rst place. 
These programs are operated with the 



mandate to minimize the inddence of 
unemployment and retain jobs by 
retraining employees who are at risk of 
losing their ^bs because of changing skill 
requirements in the workplace. 

The increased emphasis on business 
retention and employee retraining is a 
response to a broad array of ireiustry and 
labor force changes. It is also a response to 
the growing demands of in-state 
businesses for the same types of training 
assistance provkied to new and exj^nding 
businesses. 

Growing business interest in retraining 
programs and the resulting demawis for 
more public funds liave raised important 
policy questions about the role and 
effectiveness of state financed, 
workplace-based retraining r.ograms in 
state economic developmei t efforts. 
Critics of these programs argue that there 
is no clear policy rationale and intervention 
model that explain how these programs 
will retain jobs and minimize 
unemployment They argue tiiat there is 
no foni»l evaluation system for assessing 
whether retraining is effective in reaching 
th^ objectives. In addition, they argue 
that training funds are given to companies 
that wouUJ have retrained their employees 
without government assistance. In other 
v/ords, public funds merely are 
substituted for company funds which 
would have been spent for retraining even 
without the involvement of state 
programs. 



State-Hnamisd R^aining Pn^gj^iuns 



As state-financed, workplace-based 
retraining progFants grow and mature, it is 
reasonable to assume that ibese policy and 
evaluation issues will become more 
important to state policy-makers. It is 
likely that in the n^r future state agendes 
will be required to evaluate the 
effectivene^ of their training investments 
in reducing unemployment and retaining 
jobs. 



Phase One 



The first objective of th^s project was to 
clarify the policy rationale and 
intervention model for state-financed, 
workplace-based retraining programs in 
terms of their twin objectives of retaining 
jots and preventing unemployment. The 
second objective was to develop an 
evaluation system for these programs that 
was comprised of: 

• Business screening guidelines for 
tai^eting training investment and 
minimizing substitution effects. 

• PerfomanM outcome svstem for 
measuring program outcomes that 
are necessary to achieve these two 
program objectives. 

In phase one, the project developed a 
policy rationale and intervention model for 
these programs based on a review of two 
programs-California's Employment 
Training Panel and Illinois' Prairie State 
20(X) Authority. Ideally, state-financed, 
workplace-based retraining programs are 
designed to reduce unemployment and 
retain jobs by: 



• Improving business performance 
and thecompetitlve standing of the 
busin^^ within its iiKlustry. 

• Providing marketable skills to 
retrained workos. 

Because of their strong economic 
developnumt focus, these progran^ put 
their major emphasis on improving 
business performance. They are based on 
the assumption that employee retraining 
reduces the incidence of unemployment 
and retains jobs by improving the 
productivity of workers. In turn, worker 
productivity is a function of both worker 
skills and the integration of theseskills into 
a fooductive workplace that incorporates 
appropriate process technologies, job 
design, labor-management relations, 
compensation systems, and employment 
security policies. To be successful, 
employee retraining must be integrated 
with larger workpbce changes that are 
intended to improve work unit or 
company performance. 

However, state-financed, workplace- 
based retraining programs, as employment 
and training programs, should minimize 
the likelihood of future unemployment of 
retrained workers by helping them gain 
marketable skills. In some cases, retraining 
projects will not be associated with 
improvements In business perfonnance. 
By establishing the attainment of 
marketable skills as a condition, state 
programs provide retrained workers with 
an additional assurance that future 
unemployment will be minimized as a 
result of their participation in the 
retraining pro^t- 
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In rombinlngboth company pwforinance 
and woriteis' ddlls as basic cordWons, a 
successful training i^oject is expected to 
result in the following performance 
outcomes (see Figure 1, Chapter 2): 

• Ttainin y (Beh ^rinrap OhfecriveS. 
Retrained workers have attained 
behavioral skill objectives that are 
designed to improve woric unit or 
company performance and 
enhance the employment 
opportunities of workers out^e 
the company. 

• Wnrk Unit Performance. Worker 
retraining is associated with 
imx^roved pcrfoimance of the work 
units that participated in the 
training project. 

• Company Performance. Worker 
retraining and associated 
Improvements in work unit 
performance are related to 
improvements in company 
performance. 

• Trainee Earnings. Worker 
retraining results in stable or 
improved employment and 
earnings for retrained workers. 

Based on this intervention model the first 
phase of the project also developed 
business screening guidelines that wouki 
maximize the effects of training projects on 
program objectives and minimize 
substitution problems. The screening 
guidelines were based on the assumption 
that retraining projects will have their 
greatest effect when: 

• Worker retraining is integrated 
into larger company plans to 



improve company performance 
and maintain or enhance its 
competitive standing within its 
industry. 

• Worker retraining isa critical factor 
in improving company per- 
formance because of significant 
changes in skill requirements. 

• Retraining provides workeis with 
transferable skills that are likely to 
be in high demand in their 
respective labor markets. 

• Businesses are unlikely to 
ureiertake the retraining without 
outside intervention such as 
government assistance. 

Seven screening guidelines wert 
devdoped from tiiese assumptions. As 
shown In Figure 2 (Chapter 2), tiiese seven 
guidelines included three guidelines 
addressing company strategies and 
performance objectives and skIU 
requirements. The remaining four 
guidelines addressee substitution 
screening. 

These business screening guideltoes and 
the four perfonnance outcome objectives 
constituted the formative evaluation 
modd tl«t was assessed in phase two of 
the project. The evaluation model is 
presented in Figure 3 (Chapter 3). 



Phase TWO 



The secoiKi phase of the study addressed 
the feasibility of Implementing the 
evaluation model In four state-flnaiKed, 
workplace-based retraining programs. 
These programs were: (1) CalifonUa's 
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Employment Training Panel (2) Illinois' 
Prairie State 2000 Authority, (3) Missouri's 
Customized Training Program, and (4) 
New York's Economic E>evelopment Skills 
Training Program. The fearfbility study 
was based on 24 case studies of training 
pro^cts funded by thefourstateprcgrams. 
The study also involved x^ted leseaxch 
activities on business training and 
evaluation practices and the availability of 
necessary industry and labor market 
information. 

This second phase addressed fbiu* major 
feasibility questions. The findings on each 
feasibility questionare summarized bdow. 

• Business Training Design and 
Evaluation. Is the system 
compatible with business practices 
in training design and evaluation? 

The proposed evaluation system is 
consistent with widely recognized models 
of business training design and evaluation 
for training projecte that are intended to 
improve business perfonnaiKe (see Figure 
4, Chapter 4). The evaluation system is 
consistent with what many training and 
development professionals recommend is 
the most appropriate way for businesses to 
plan and evaluate their own training 
investments. The only major diflerence 
between the evaluation model and 
recommended business practice is the 
model's emphasis on trainee earnings. 

• Definition and Measurement of 
Performance Outcomes. Can the 
four performance outcome 
indicators (company performance, 
work unit perfonnaiKe^ training 
objectives, and trainee earnings) ]x 
measured in a valid and reliable 
manner that is consistent with 



business performance measures 
and data collection activities? 

The case sttidira of 24 training projects in 
the four state programs suggest that 
company or work unit performance 
outcome indicators can be easily identified 
and measured in valid and reliable ways 
that are consistent with the methods that 
businesses use to m^sure their own 
performance. Consequently, these 
company or work unit performance 
measures should not impose extra costs on 
businesses because businesses normally 
collect the same inforaation for theJr own 
internal puipc^. 

Also, we conclude that company and 
work unit {^rfomunce are best measured 
in terms of operational objectives and 
performance targets that are logically 
linked to business strategies and 
perfbnnance goals (see Figures 5 and 6, 
Chapter 4), The case studies identified 
seven mapr types of operational objectives 
and performance targets that could be 
linked to competitive strategies and 
business goals as well as financial and 
market performance. 

The definition and measurement of 
training objectives varied widely in the 24 
case studies. However, we conclude that 
aU 24 companies could have restructured 
their training curricula in terms of 
measurable training or behavioral 
objectives. Although this requirement 
wouW require additional effort from most 
companl^, it is consistent with widely 
accepted standards of business training 
design. The benefit of this additional effort 
is more effective training projects. 

Because of validity and reliability 
problems in employee testing, we conclude 
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post-t»ting requirenwnts should be 
established. They should be simple and 
easy to measure and should address 
b^vioial skills ditectly related to a core 
8si of tndidng objectives. 

The use of trainee eandnp as a ma^r 
performance outcome indicator has 
signiikant problems because of ibe wide 
variation in company compensation 
systems. It still is useful to track and report 
the employment and earnings experiences. 
Data collection costs for businesses as well 
as state programs can be minimized by 
using Unemployment Insurance (UI) 
wage records. 

• Business ScreeninK Guidelines. 
Can the evaluation system 
practically address targeting and 
substitution problems through a 
set of business screening 
guklelines? 

Other substitution problems can be 
addressed through state contractual and 
in-kind policies that pay for the type of 
training that companies ordiiuuHy do not 
undertake on their own initiative. 
Typically, these training activities include 
classroom and laboratory training and 
highly stmctured on-site training. 

The seven screening guidelines can 
effectively address the most important 
targethig and substitution problems faced 
by state programs. Based on the 24 case 
studies, we found that companies can 
dearly statecompetitivestrategy (e.g./COSt 
quality/ differentiation), performance 
objectives, and the need for relraining. In 
most cases, they can cite industry 
benchmarks that they must m^ to remain 
competitive. Industry information on 
competitive benchmarks is available to 



state programs. They may use this 
infonnation to woric with companies in 
defining meaningful performance 
objectives. 

In addition, most companies can define 
training objectives and retraining 
requirements. The only poss&le exception 
is some small companies. State programs 
can establish simple and inexpensive 
procedures to establish the irtarketability 
of the skills acquired through training 
projects. Although the substitution 
guidelines are more difficult to implement, 
they will be effective in rediHiing some 
substitution problems faced by state 
programs. 

The seven screening guidelines can be 
Incorporated effectively into a simple 
application process that would not require 
substantial&ctraworkfbr either businesses 
or state program staff. The application 
process and the screerting guidelines are 
summarized and illustrated in Figure 7 
and 8 (Chapter 5). 

• Implementation of foe Evaluation 
System. Can the evaluation 
system \x implemented effectively 
given the administrative budgets 
of state programs and the 
background aiKl training of staff? 
Would businesses continue to 
participate in the program? 

The proposed evaluation system can be 
administered effectively without 
substantial additional costs being incurred 
by either the business or the program. In 
order to implement the system, additional 
staff training will be required. This staff 
training will be necessary in order to 
standardize the method for reporting 
business performance objective. Most 
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state staff are trained sufficiently in 
identifying dear training objectim. 

Future business partkipation is difficult 
to assess. However, no business voiced 
objections over specifying business 
performance objectives or training 
objectives in future proposals. We propose 
a simplified application process where 
^ch screening and performance outcome 
issue can be addressed ^Kavely without 
causing a significant increase in the 
requirements of the current application 
proceduies now utilized by any of the fbur 
slate programs participating in the project. 

This feasibility study resulted in sbc major 
condusionsontheeffectsofstate-financed, 
workplace-based retraining programs: 

l State trai^ang grants generally will 
expand the scope, shorten the timing, and 
enkrge the number of workers who 
partkdpate in training projects. 

2. Training projects were most effective 
when they tied dearly to specific business 
goals and performance objectives and 
when the training plan defined clear 
training or behavioral objectives. 

3. State traintag grants have an important 
value in overcoming management 
uncertainty over the importance of 
retraining workers for the introduction of 
new technotogies and work processes. 

4. Substitution risks are lowest when 
state grants concentrate on the direct 
instructional costs of classroom and 
laboratory training. 

5. In many cases/ company management, 
especially in small businesses, was 
uiKertain about how to utilize training as 



an effective agent of strat^ d»nge to 
improve company performance. 

6. State programs may play a major 
catalytic rote in emrouraging businesses to 
use training as an agent of strategic change 
and estaUi^ permanent learning systems 
within compaiiies. 

These major findings and condusions 
from the feasibility study provide the basis 
for five recommendations. 

2. States should adopt the business 
screening and performance outcome 
evaluation system as the basis for a more 
detailed formative evaluation system 
worked out by each state in conformance 
with program statutes, policies, and 
contracting practices. 

2. States should ^tablish multi'Stage 
projects that re&fgnixe that training may 
htvohfe multi^ear efforts. However, each 
stageskould have intermediate trainbtgor 
performance objectives with all stages 
justified in Urms affinal business goals 
and performance objm:tives, 

3. States should establish special 
technical a^istance programs Uuit would 
improve access for small businesses to 
retraining programs. These programs 
should provide technical assistance in 
defining competitive strategies and 
performance objectives as well as training 
objectives and training design. 

4. State programs should implement the 
formative evaluation system over a 
two-year implementation period. 

• Year 1 - Implementation of 
business screening and 
performance outcome system with: 
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a) an evaluation of the effects of 
system implementation on 
program operation including 
business participation, application 
screening, and contract 
mana^ment^ and b)case studio of 
a reprraentative sample of training 
projects on company perfonnance 
and training objectives. 

Year 2 - Refinement of business 
screening and performance 
outcome system with a formal 
follow-up study of a representative 
sample of training project on 
company performance, training 
objectives, and trainee earnings 
based on unemployment insurance 
(UD wage records. 



• Year 3 -Implementation of the full 
follow-up system based on a 
representative sample of training 
projects and the implementation of 
a system to track trainee earnings 
based on Unemployment 
Insurance (UI) wage records. 

5. Federal tmd staU gavgrnmm^ should 
coopemte in establishing a resource center 
for disseminating information on 
state-funded training projects and their 
role in improving ^ competitioeness of 
industry. The resour^ txnter also should 
provide models of comprehensive training 
systems ^at conibhte basic, occupational 
and job-specific training. 
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Chapter One 



CHAPTER 1 
INTRODUCTION AND OVERVIEW 



Since the 1960s, states have created 
customized training programs as part of 
their economic development efforts to 
attract, expand, and retain businesses. The 
purpose of these programs has been to 
expand and retain employment in states by 
assisting businesses in training and 
retraining their workers. Unlike 
traditional employment and training 
programs, which focus on individuals and 
specific populations, state-financed 
customized training programs are mainly 
economic development programs. They 
focus on the training needs of business and 
the use of state funds to increase and retain 
jobs for the state. Recent studies estimate 
that 45 states now have one or more 
customized training programs (Stevens, 
1986; Ganzglass and Heidkamp, 1986; 
American Society for Training and 
Development, 1989). 

Stevens (1986) observed that 
state-financed customized training 
programs emerged in the 1960s and 1970s 
as part of state efforts to compete with 
other states for business inv^tmenL hi 
most states, they were established as 
highly flexible, streamlined programs that 
offered financial ira:entives to businesses to 
locate in the state or to expand 
employment. They were also established 
to reduce skill shortages and address 
training needs that public educational 
institutions could not respond to quickly 



enough. As a result, these programs have 
very broad economic development 
objectives and funding priorities that 
continue to evolve according to changing 
economic development priorities and 
in-state business demand for training 
assistance. 

Although state-financed, customized 
training progian^ are similar to each other 
in their strong economic development and 
busine^ focus, they differ significantiy in 
their emphasis on busix^ss attraction and 
expansion (i.e., training newly hired 
workers) versus business retention (Le., 
retraining employed workers). 
State-financed, workplace-based 
retraining programs-programs that focus 
most of their resources on business 
retention and retraining-are a relatively 
recent phenomenon. However, since the 
early 1980s, business retention and 
retraining have become a significant 
component of most state customized 
training programs (Stevens, 19^; ASTD, 
1989). 

This increased emphasis on business 
retention and employee retraining appears 
to be a response to a broad array of 
economic, demographic, and labor force 
changes that include an aging workforce, 
increased national and international 
competition, rapid changes in production 
and information technologies, chronic 
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problems in worker dislocation, and 
lagging private sector investment in 
human resource development. Such 
programs also are a response to the 
growing donands of in-state businesses 
for the same types of txaining a^istance 
given to new and expanding business^. 

Growing bu^ess interest in retraining 
programs and the r&sulting demands for 
more public funds have raised important 
policy questions about the role and 
effectiveness of state-financed, 
workplace-based raining programs in 
state economic development efforts. 
Currently/ these programs are justified on 
the grounds that they minimize the 
inddeiKe of imemployment and retain 
jobs by retraining employees who are at 
risk of losing their jobs because of 
changing skill requirements in the 
workplace. Critics of these programs have 
argued that there is insufficient evidence to 
support the daim that retraining is an 
effective strategy for preventing 
unemployment and retaining pbs. Many 
critics are skeptical of the claim that 
companies woidd not have retrained their 
employees without government 
assistance. Instead, they suggest that 
public funds are merely substituted for 
private funds. 

As state-financed/ workplace-based 
retraining programs grow and mature, 
these policy and evaluation issues will 
become more important to state policy 
makers. Already, state agencies have been 
asked to evaluate the effectiveness of their 
training investments in reducing 
unemployment and retaining jobs. 



Project Backgnnind and 
Study Approadi 

In 1987, California's Employment 
Training Fanel and Iliinois' Prairie State 
2000 Authority identified a common 
interest in developing a formal evahiation 
system for their retraining projects. Like 
other retraining programs, these two 
agei%:ies had worked to clarify funding 
priorities and develop indicators of 
program performance. However, these 
agencies recognized that they would 
require a more fully devdoped evaluation 
system in the future. 

Based on the evaluation issues raised by 
these agencies, the National Govconors' 
Association oiganized a study to develop 
and assess an evaluation system for 
state-financed, workplace-based 
rebaining programs. Through funding 
from the National Commission for 
Employment Policy, the National 
Governors' Association sponsored a 
two-phased project. The first phase was a 
study of the policy and evaluation issues 
for state-finaiKed retraining programs and 
the development of an evaluation 
approach that could be assessed and 
refined through a comparative state study 
of retraining projects. This resulted in a 
report entitled State-Financed, 
Workplace-Based Retraining Progmms 
(Creticos and Sheets, 1989). This report 
proposed a fonnative evaluation approach 
that was consistent with the policy 
rationale and program objectives of 
state-financed workplace-l^ed retraining 
programs. 
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The second phase of the project was 
designed as a feasibility study of the 
evahiation approach based mainly on case 
studies of 24 retraining projects in four 
state programs (see Api^wdix A). The four 
state programs participating in the second 
phase were: 

• California's Employment Training 
Panel 

• Illinois' Prairie State 2000 
Authority (Employer Training 
Assistance Program) 

• Missouri's Customized Training 
Program 

• New York's Economic Develop- 
ment Skills Program 

Although the missions and objectives of 
these programs arequite similar, they vary 
considerably in scope, size, funding 
priorities, selection criteria, and 
organizational structure. Brief 
descriptions of these state programs are 
provided in Appendix A. 

The feasibility study was designed to 
address two important components of an 
evaluation system: (1) business selection 
and screening guidelines; and (2) 
performance assessment based on four 
types of perfonnance outcome measures: 
company performance, work unit 
performance, skill competencies, and 
wages and earnings. 



Feasibiliiy Issaes and 
Stmcfane of flieRepoit 

The feasibility study was designed to 
answer four basic questions about the 
proposed evaluation system. 

• Business Tr a»^i«B Deai^ and 
Evaluation. Is the evaluation 
system compatible with bu^ess 
practices in training design and 
evaluation? 

• pgfinitton and Measurement,^ 
Performancg Outcomes. Can the 
four performance outcome 
indkiators (company performance, 
work unit performance, training 
objectives, and wages and 
earnings) be measured in a way 
that is consistent with business 
performance measures and data 
collection activities? 

• Ruaingas S election and Screening 
Guidelines^ Can the evaluation 
system address targeting and 
substitution problenw in a practical 
way through a set of business 
selection and screeiung guidelines? 

• Im plementari nn Problems of tiie 
Evaluation System. Can the 
evaluation system be implemented 
in a cost-effective way given 
administrative budget constiaints 
and staff training requirements? 
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Would businesses continue to participate 
given the additional reporting 
lequixements of the evaluation system? 

In Chapter 2, we summarize the busixffiss 
screening and performance outcome 
evaluation system proposed in the 
feasibility study. The third chapter 
addresses the compatibility of the 
proposed evaluation system with good 
business practices by comparing it to a 
generally accepted model of training 
design and evaluation for business 
performance improvement The fourth 
chapter addresses the feasibility of 
developing measures for the performance 
outcome indicators. It also addresses 
whether these measures are compatible 
with business meas\iresaxKi data collection 
activities so that progiams avoid imposing 
additional costs on participating 



businesses. This chapter uses illustrations 
from the 24 case studies of training 
projects. The fifth chapter addresses the 
process monitoring questions of business 
selection and screening. The Hnal 
feasibility question on implementation is 
addressed in the sixth chapter which 
focuses on state monitoring activities. The 
seventh chapter summarizes the ma|or 
fiiKJings and recommendations from the 
feasiMlity study. 

This report is based in part on the 24 case 
studies of training pro^cts funded by the 
four state programs. These case studies are 
presented in the second vohime of this 
report, entitled. Evaluating State' 
Financed, Workplace-Based Retraining 
Programs: Case Studies of Retraining 
Projects. 
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CHAPTER 2 

THE EVALUATION MODEL: 
A Proposed Business Scieening and 
Perronnance Assessment System 



The first question addressed in this 
project was whether it was possible to 
devtelop a formative evaluation system for 
state-financed, workplace-based 
retraining programs. Because these 
programs are mainly economic 
development programs fliat focus on job 
retention and the prevention of 
unemployment, traditional evaluation 
models do not apply. In additioiv state 
economic development programs 
generally Itave not developed evaluation 
sjrstems that assess the ultimate sucx^ess of 
their efforts in retaining jobs. 

In this section, we discuss the evaluation 
system that was developed for the 
feasibility study. This modd represents a 
general evaluation approach to 
performance outcomes that is consistent 
with the policy rationale and objectives of 
these programs. The foundation of the 
system is a formal intervention model and 
a core set of performance expectatiox^ that 
establish whether the program has been 
successful in retaining jote and preventing 
unemployment. It also includes bu^ness 
screening guidelines that are designed to 
define and support these performance 
expectations as well as minimize 
substitution problems. This chapter first 
reviews the general policy and evaluation 



issues considered in developing the 
evaluation model Then, it disaisses the 
major program objectives and the 
intervention model used to define 
performance outcomes. Next is a 
discussion of the targeting and 
substitution issues that framed the 
development of the business screening 
guidelines. The chapto* concludes with an 
overview of the proposed business 
screening and performance outcome 
system for state-financed, workplace- 
l»sed retraining programs. 

Evaluation Systems: Process 
Monitann^ Pezfonnance 
Assessment and Impact 
Evaluation 



A compreh^ive evaluation system for 
state-financed workplace-based retraining 
programs should consist of three types of 
evaluation activities. 

• Process Monitoriiiit addresses 
whetho: programs are operated in 
accordance with program 
objectives and intervention models 
that are assumed to produce the 
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desiied program effects. Process 
monitoring determines whether 
the program is serving the most 
appropriate elicits targeted in the 
intervention model and whether 
the pn>gram has implonented the 
intervention model consistently 
with these cUaits. 

• Performance Assessment 
addr^ses whether programs have 
achieved the spedfk intermediate 
and final outcomes that are 
specified in the Intervention 
model. These outcomes are 
assimied to be the gross outcomes 
that are necessary to produce the 
desired program effects. 
Performance assessmoit is 1»sed 
on process monitoring activities 
that determine whether the 
intervention model was 
implemented fully with respect to 
the targeted client population. 

• Impact Evaluation addresses 
whether programs have had a 
significant effect on program 
objectives. Impact evaluation 
assesses the validity of the 
intervention model; le., whether 
perfomuince outcomes actually are 
linked to the desired effects over 
and above the effects of other 
factors. Impact evaluation is based 
on both process monitoring and 
performance assessment. It 
assumes that the intervention 
model has been implemented as 
designed and that the basic 
performance outcome are being 
achieved. 

Each activity is a necessary precondition 
for the next. The impact of the program Is 



a function of specific pro^ outcomes; the 
aggregate efiect of theseoutcomes is linked 
by theory to changes in measures related to 
the broad policy goals of the program. In 
order to determine whether the 
intervention model is a factor in achieving 
the outcomes that are observed later, it is 
essential to monitor the implementation of 
the model 

The first step in developing a 
comprehensive evaluation system for 
state-financed, workplace-based 
retraining programs is to clarify tlw major 
program objectives and intervention 
model and establish the major 
performance outcome indicators. The next 
step is to establish business screening 
guidelines that support the 
implementation of the intervention model 
and performaiKre outcomes and that a^t 
programs in selecting businesses with 
which the program wiU have the greatest 
impact. Then, state programs can shift 
their attention to develoj^g an impact 
evaluation approach that will evaluate the 
intervention model. 

Performance assessment and impact 
evaluation are designed to answer two 
different types of questions about the 
effects of public programs. Impact 
evaluation addr^s^ the most important 
and fundamental question. For 
state-financed, workplace-based 
retraining programs, impact evaluation 
determines whether a program had a 
significant effect on retaining business and 
reducing unemployment over and above 
the effects of other factors including 
organizational and technological changes. 
Also, an imjact evaluation will determine 
whether these changes would have 
happened in the absence of the state 
intavention. 
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In contrast, perfonnance ass^ments 
result in statements about whether 
progFam interventions are tssociM^ unth 
sigxUficant changes in the intennecUateand 
final performance outcomes that are 
asstuned to result in positive impacts on 
job retention and the reduction of 
imemplojonent 

Although impact evaluation provides a 
moredirect and fundamental evaluation of 
public programs, it is very difticult to 
design and implement even in traditional 
employment and training programs. 
Impact evaluation is particularly difficult 
in innovative public programs undergoing 
continual change due to the fonnalization 
of program ob^ctives and the growing 
sophistication of operating agencies. 

The problems of conducting impact 
evaluations for state-financed, 
workplace-based retraining programs are 
summarized in Appendix C. The main 
body of this report focuses on performance 
assffismentand process monitoring, which 
directly support a performance 
assessment system and address major 
substitution problems. 

Frpgram Objectives, 
Intervention Model, and 
Perfonnance Outcomes 

State-financed, workplace-based 
retraining programs are designed to 
reduce unemployment and retain jobs by 
retraining employees at risk of losing their 
jobs because of changing skill 
requirements. Such programs assimie that 
retraining reduces unemployment and 
retains jobs by improving the productivity 
of workers. In turn, worker productivity is 



a function of both worker skills and the 
Integration of these skills into a productive 
workplace that incorporates appropriate 
process technologies, good job design, 
good management-labor cooperation, 
appropriate compensation systems, and 
credible employmoit security poUdes. To 
reduce unemployment and retain jobs, 
retraining must be integrated with laiger 
workplace changes that are interaied to 
improve work-unit or company 
pofonnance. 

State-financed, workplace-based 
retraix\ing programs should reduce the 
likelihood of future unemployment of 
retrained workers by helping them gain 
marketable skills. By establishing the 
attainment of marketable skills as a 
condition, the state efliectively plays to its 
historical strength in designing and 
operating employment and training 
programs. As a consequeice/ in order for 
an outcome to be considered successful, it 
would not be necessary either for a 
business to improve its competitive 
position or for its work imits to improve 
their performance. It would be sufficient 
to show tluit the retrained workers were 
more valuable in the marketplace as a 
consequence of the skills they gained 
through the program. 

As shown in Figure 1, taking both 
maximum and minimum conditions 
together, a sucx^ful ieb:aining project 
should result in the following performance 
outcomes: 

• Attainment of Skills Training 
Objectives. Retrained workers 
have attained specific skills 
training objectives that are 
deigned to improve work unit or 
company performance and 
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enhance the employment 
opportimities of workers outside 
the company. 

• Work Unit Perfotmancc. Worker 
retraining is associated with 
improved perforanaiwe of the work 
units within the company that 
partkipated in the training. 

• Com pany Performance. Worker 
retraining and associated 
improvements in work unit 
performance are related to 
improvements in company 
performance and the overall 
competitive standing of the 
company within its industry. 

• Worker EarninES. Worker 
retraining results in stable or 
improved employment und 
earnings for retrained workers. 

These four performance outcomes 
provide the foundation for the proposed 
performance assessment system for 
state-financed, workplace-based 
retraining programs. In order to 
implement the intervention model these 
measures also require screening guidelines 

to Insure that participating comjanies can 
define their projects in terms of these 
outcome 

Busing Screening and 
Substi tution Minimization 

Targeting is the most difficuh question 
faced by state-financed/ workplace-based 
retraining programs: i.e., how to select 
businesses for financial assistance so as to 
maximize the impact of the program arul 
to improve the overall health of the state 
economy. Program impact is a function of: 



(1) the importance of the bu^ness to the 
state economy, (2) the importaitte of the 
training project to the retention of 
bu^ess'sor jobsinthestotei^andO) the 
probability that the company would not 
undertake the training project without 
state financial assistance. 

In proposing possible state program 
felines and administrative procedures 
for the targeting question, three major 
evaluation issues were centered: key 
industry /business identifkation, burfness 
screening, and substitution minimization. 
As discussed in the first report of this 
pro^ (Creticos and Sheets, 1989), the 
targeting of key imiustries or businesses 
shouki be based on the unique economic 
structure of states, their overall economic 
development strat^es, and the funding 
priorities and activities of other stete 
agencies. As a result, individual states 
shouki dedde which industry sectors (e.g., 
manufacturing versus nonmanufacturing) 
or which types of businesses (e.g., small 
versus large businesses) should receive 
funding priority. This first targeting issue 
is beyond the scope of this study. 

However, state programs can establish 
guidelines to help them select businesses 
within a targeted population whose 
participation in the program is most likely 
to have a positive effect on program 
objectives. These guMelines address two 
basic questions: (1) how does an agency 
screen and select those companies that are 
most likely to use state trailing funds to 
achieve the performaiKe outcomes that are 
assumed to result in job retention and the 
prevention of imemployment, and (2) how 
do& an agency screen and select those 
companies that would not do the 
retraining in the necessary scope and 
timeframe without sUte assistance? 
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The Intervention model shown in Figure 
1 is based on theassumptionthatictraining 
projects will have their greatest effects in 
the following circumstances: 

• Worker retraining is integrated 
into a company's plans to improve 
its performance and enhance its 
compeHtive standing within its 
industiy. 

• Woricerretminingisacritkal factor 
in improving company per- 
formance because of significant 
changes in skill requirements. 

• Worker retraining is integrated 
into a larger action plan (ircluding 
technological and organizational 
changes) that is implemented 
simulUneously with the txaining 
program. 

® Retraining provide workers with 
transferable sldUs that are likely to 
be in demand in state or local labor 
markets. 

• Companies are unlikely to 
undertake the retraining without 
outside intervention such as 
government assistance. 

These assumptions provide the basis for 
seven business screening guidelines. 
Three guidelines address the likelihood 
that the training project will have a 
significant effect on the major performance 
outcomes. Four address the likelihood 
that these effects can be achieved without 
major substitution problems. These seven 
guidelines are presented in Figure 2 and 
described below. 

• Competitive Strategy, Per- 
formance Objectives, and 
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Industry Benchmarks addresses 
whether businesses have linked 
their training projects to sirategies 
and objectives that will make them 
more competitive in their 
industries relative to industry 
benchmarks or competitive 
^ndards. This guideline requires 
that businesses describe how 
state-financed training will be 
used to achieve measurable 
outcomes in company or work- 
unit performance that will have a 
materia! effect on their competitive 
standing. 

• Identification of New Skill 
Requirements addresses whether 
businesses have defined new 
functional skills for improving 
company or work-unit 
performance that will lead to 
retaining the jobs of retrained 
workers. This guideline requires 
that companies describe such skills 
and link them to performance 
objectives. It supports the 
definition and measurement of the 
performance outcome addressing 
training or bel» vioral objectives. 

• Skill Shortages in State addresses 
whether the training project will 
provide retrained worker- with 
marketable skills that could 
increase their chances for 
reemployment if the business is not 
successful in improving its 
performance and either closes or 
reduces the size of its workforce. 
This guideUne is used to estimate 
the likelihood of reemployment if 
the first guideline is not successful 
in selecting those businesses that 
are highly likely to improve their 
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Figure 2 

Evaluation Model for State-Financed 
Workplace-Based Retraining Programs 
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competitive standing through the 
training pro^. 

• Financial Need Assurance 

addresses whether a company can 
show that it do^ not have the 
r^urces necessaiy to conduct a 
training project of the scope and 
within the time necessary to 
achieve its perfonnance objectives. 
This guideline addresses the 
problem of direct and simple 
substitution of public for private 
training funds. 

• Transferable Skill TraininR 
addresses whether a company is 
likdy to undertake training in light 
of the substantial inv^tment it 
would be required to make in 
transferable (e.g., basic and 
vocational) skills that are in 
general demand in the labor 
market. This guideline attempts to 
assess wheth^ state assistance is 
required to offset a company's risk 
of losing its training investment to 
ether companies. It also assess^ 
the degree to which the trainhig 
project involves basic and 
vocational skill training that couki 
alleviate a skiUs shortage in the 
state. This guideline is similar to 
the third gukleHne except that it 
focuses more directly on the 
sulstitution question. 

• Previous Training Expenditure 
addresses whether the business has 
undertaken previous training 



projects with similar content and 
similar types of workers. This 
guideline is based on the 
assumption that substitution is 
greatest when companies have 
undertaken similar projects before 
with their own resources. 

• Disinvestment Decisions 
addresses wheth^* a company is 
requesting training assistance 
because of potential disinvestment 
and /or layoff decisions from a 
parent company. This guideline 
addresses situations in which a 
business establishment is under 
pressure to improve its 
performance or competitive 
standing in order to remain open 
but is unable to secure corporate 
financing to retrain its workers at 
the necessary level and within the 
necessary time. 



Summaiy and Condiisions 

The proposed evaluation system for 
state-financed, workplace-based 
retraining programs is shown in Hgure 2. 
The proposed business screening and 
performanceassessmentsystem consists of 
seven major bu^ess screening guidelines 
and four performance outcome measures. 
The following chapters ofhr a detailed 
discussion of the findings and conclusions 
from the feasibility study. 
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CHAPTERS 

BUSINESS TRAINING DESIGN AND 
EVALUATION: Models of Training for 
Ferfoimance Improvement 



The first major fea^bility question is 
whether the proposed evaluation system 
for state programs is consistent with the 
ol^ctives of private sector training and 
cost-effective training design and 
evaluation. A state evaluation system 
should be consistent with how bu^esses 
look at their own investments and with 
what ^tiey assume to be necessary planning 
reqtiirements. This correspondence to 
private-sector practice is important for two 
reasons. First, an evaluation system 
should not discourage or distort business 
participation in the program. 
State-financed retraining programs are 
discretionary grant programs tluit depeiKi 
on voluntary busing participation. If 
businesses are asked to submit training 
proposals that address issues only weakly 
related to private-sector objectives and 
planning requirements, qualified 
companies prol»bly will be discouraged 
from participating and the transaction 
costs for the business will increase. 
However, if the evaluation system is 
consistent both with business ob^ves 
and with exemplary training practices, it 
will contribute to business understanding 
of the Unkage between training and the 
improvement of internal operations while 
it acts as an effective screening device for 
the program- 



Second, if the evaluation system is 
inconsistent with business objectives and 
practices, it may distort the original 
ob^ves and strat^es of the training 
pr^ect and actually work against both the 
bu^ness ami the program. It isa common 
problon in both business aad government 
that what we measure and get rewarded 
for is what we pay attention to. If the 
evaluation system the company to 
measiu« and track information that is 
unimportant to the purpose of the training, 
this information may gain sufficient 
importance to the buMn^ to undercut the 
effective implementation of the project 
The proposed evaluation approach and the 
interview instruments for the case studies 
were based on models of busiiwss training 
design and current practices in business 
evaluation. 

1)31x111^0^^ for 
Ferfoimance Improvement 

Increasingly, business training 
departments are asked to support specific 
business strategies and company 
performance objectives. Employee 
training thus becomes a strat^c agent for 
change in businesses undergoing major 
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organizational and technological changes 
in order to achieve specific perfoixnance 
improvements (Ghezson and Moote, 1987; 
Brinkerhoff. 1989). One wkiely accepted 
model of training design for pafonnance 
improvement is summarized in Figure 3 
and described below (Camj^ell, 1989). 

Business Strategy and Performance 
Objectives. The first st^ in designing a 
training project is to understand the 
compSiiy's perfoimance objectives. Are 
they dearly stated and measurable and are 
they consistent with tl» overall busii^ss 
strategy? Performance objectives should 
be written down and agreed upon for each 
work unit in which the training and 
development effort is taking place. 

Organizational and Technolottical 
Changes, Tob Redesign and Training 
Objectives. The secord step of the training 
modU is defining the training or 
behavioral objectives of the newly 
designed jobs. In simplest terms, training 
objectives are the functional skills or job 
tasks that the trainee should be able to 
perform after finishing the program The 
redesigned jobs should be dkectly relevant 
to the performance objectives of the work 
unit involved in the training program. 
This is to insure that the planned 
performance outputs of a trained 
employee are consistent with and 
important to the performance objective of 
the work unit. 

Next comes the identification of the 
training objectives. These objectives 
include the set of clearly defined functional 
tasks and behavioral skills that are 
required of an employee in the redesigned 
job and that must be learned by a 
significant percentage of the employees of 
the work ui^t. 



The final consideration involves 
kier tifying the proficiency levels required 
for eadi training or behavioral objective. 
These levels provide the basis for the 
training departaient to determine whether 
the trainees have been successful in 
reaching the training ob^ves. 

Training Content; Knowledge and 
Skills. The third step in developing a 
training pro^ Is to specify the knowledge 
and skills that will permit competent 
performaiKie of the ob^ves. This step 
first involves a definition of the specific 
training modules needed for each level of 
knowledge or skill required to meet the 
training objective. This is followed by 
decisions about the proper sequence of 
these modules in the curriculum. 

Training Methods. Media, and 
Conditions. The fourth step is to 
determine the methods, media, and 
learning conditions most appropriate for 
each training module to achieve the 
objectives of the training project. 
Instructional methods and media shouk) 
be selected based on the level and type of 
knowledge or skill to be produced. For 
example, knowledge of key terms may be 
learned most effectively through 
classroom lectures supported by drill and 
practice exercises through computer-based 
instruction. In contrast, a problem-solving 
skill may be taught better through group 
discussions and laboratory exocis^ that 
call for the desired cognitive and motor 
skills. 

Leaning conditions can influence the 
interest, involvement, and conunitment of 
the trainee. Motivation increases when the 
trainee understands the purpose of the 
training in terms of valued outcomes such 
as job security, moi. y, professional 
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Figured 

lYninlng Design for Strategic Change and 
Business Performance ImproinNnent 
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Did the training objectives contribute to the achtevemeat of company performance objectives? 
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opportunities, ard »lease from other paid 
responsibilities. Motii^tion aiKi worker 
commitment may be affected by other 
factors such as type of trainees, training 
facility, training schedule, aitd learning 
recognition. 

Assessment of Individual Diffeiences. 
The fifth step is to assess individual 
differences that may affect trainee 
performance. The &K)st crudal factor is to 
identify difference in aptitudes, prior 
knowledge and skills, and task 
understanding. These may require 
indivMualized instruction or advanced 
placement within the training curriculum. 
Another issue is to identify individual 
differences in motivation that tmy require 
alternative learning coiKlitions. In some 
cases, concerns over individual differexKes 
may give way to concerns with 
standardize! knowledgeandcommitment 
to a company training and performance 
improvement program (e.g., quality 
control) or the added ^t of testing all 
employees. In other cases, such as those 
that involve training in technical skills, 
formal testing (to detennine math and 
reading levels, for example) may be critical 
to the success of the program. 

Evaluation of Training Prolect. The final 
step is the evaluation of the effectiveness of 
the project based on criteria developed for 
the specific learning ol^jectives aiui their 
contribution to the company's 
performance objective. These evaluation 
issu^ are discu^ed l^low. 



'fiaining Evaluation in 
Business 



In conskiering tl^ compatibility of the 
proposed evaluation approach with 
evaluation practices in business, two 
critical issues should be addressed: (l)the 
typesofperfbnnanceoutcomeconsidered 
in business evaluation, aiKi (2) the linkage 
between training objectives and company 
or work-unit performance. 

Levels of Training Evaluation. Business 
evaluation of training is best tmdeistood as 
addr^ing one of four levels: (1) trainee 
reaction, knowledge and skills, (3) job 
performance, and (4) company 
performance (Kiikpatrkrk, 1982). 

Trainee reaction refers to the 
self-reported reaction of the trainee 
toward the program This information 
usually is obtained through quQ»tionx»ires 
administered after the training. 
Knowledge and skills refer to what was 
learned in specific modules of the training 
program. Trainees are asked to describe 
what they know in a written or oral test. 
They are also asked to demonstrate 
functional skills in a laboratory or a 
simulated work setting. Job performaiKe 
refers to whether the individual can 
perform satisfectorily the specific set of 
behavioral skills or functional tasks 
reqiured on the job. Job performance can 
]x evaluated l»sed on the assessments of 
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trainers in laboratory and simulated work 
settings or of supervisors or trainers 
observing actual performance on the job. 
Company performance refers to the 
operating pofbnnance of the company. It 
includes issues such as sales growth, 
profitability, market share, productivity, 
quality, and customer satisfoction. 

The most common type of evaluation is 
trainee reaction done largely through 
post-training questionnaires 
(Brandenburg, 1989). The second most 
common type is the evaluation of 
knowledge and skills usually d-^ne 
through some type of written test. A 
significant number of companies evaluate 
job performance through work simulation 
or ratin]^ of on-the-job performance. A 
large number of companies evaluate 
performance through standard sets of 
performance measure. 

Although company performance 
evaluation is the least common of all 
evaluation levels, some training and 
development professionals argue that it is 
becoming very important in companies 
because management increasingly uses 
training as a change agent and wants to 
know whether performance improved 
after training (Brinkerhoff, 1989; Swanson, 
1989; CampbeU, 1987). 

Training Objectives and Company 
Performance. Although the evaluation of 
a training project in terms of company 
performance is growing in importance, 
there still is considerable confusion over 
the methods that evaluators should use in 
examining the linkage between training 
and performance. Training professionals 
commonly identify at least four major 
determinants of human performance 
improvement. These are: (1) the 



knowledge and skills necessary to poform 
job requironents, (2) task knowledge and 
awareness, (3) basic abilities, and (4) 
motivation to learn and perform 
(Campbell 1987). 

At best, training departments have direct 
control oidy over the improvement of task 
knowledgeand theunderlying knowledge 
and skill necessary to perform the training 
or behavioral objaztives of a job. Training 
departments vary tremendously in the 
range of their control over the abilities of 
trainees and their motivation to learn. 
These foctors usually are controlled by 
other units within the organization or they 
represent short-term constraints. In 
addition, training departments rarely have 
direct control over job design and other 
organizational and technological changes 
that will be combined with the behivioral 
skills from training to achieve company 
performance objectives. 

Howevo", as presented in the training 
design model in Rgure 3„ many training 
and development professionals argue that 
training directors should take 
responsibility for securing dearly defined 
training objectives that are consistent with 
other organizational and technological 
changes (Brandenburg, 1989; Swanson, 
1989, Brinkerhoff, 1989; CampbeU, 1987). 
Training evaluation shouM not be used in 
an effort to prove that training aiused the 
performance improvement or that a 
percentage of the performance 
improvement can be attributed to training 
(Brinkerhoff, 1989). Rather, the evaluation 
questions should br. (1) did the training 
result in the achievement of training 
objectives, and (2) were these training 
obiectiyes associated with ^nd necessary for 
improvements in company performance. 



17 



State-Hnanced Retraining Programs 



This perspective is consistent with the 
evaluation objectives of the proposal 
business screening and performance 
assessment system for state-financed/ 
workplace-based retraining programs. 
This pospective emphasi^^ tl^t training 
should be connected logically to a 
performance objective. In addition, the 
implementation of the trailing project 
should be associated with the achievoo^t 
of training objectives and the ultimate 
performant^ objectives of the work unit or 
company. 



Siimmaiy and Conclusions 

The evaluation system proposed in this 
chapter is consistent with one prominent 
model of business training d^ign and 
evaluation. In general, the system 
addresses what training ai^ development 



professionals suggest is the proper way for 
businesses to plan and evaluate their own 
training investments. In addition, the 
model is applied in some leading 
businesses that use training as a change 
agent 

The only difference between the 
proposed evaluation model and cunent 
business practice is that the proposed 
model emphasizes wages and earnings 
aftertraining. This issue is e^lored in the 
next chapter. 

Another unresolved question is whether 
this business model of training and 
evaluation actually is used by the 
budnesses that normally receive tiaining 
assistance from state-financed, 
workplace-based retraining programs. 
This also is addressed in the next chapter. 
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PERFORMANCE OUTCOMES: Company and 
Work-Unit Performance, Training Objectives, 

and Worker Earnings 



Thesecond feasibility question is whether 
it is possible to develop performance 
indicators that are valid and reliable 
mea»ires of program performaiKe ifdth 
regard to company performance, work 
unit performance, skill competencies, and 
trainee earnings. A rdatai que^on is 
whether it is feasible to collect the 
necessary data at a cost that is acjccptable to 
the company and the state agency and that 
is lower than the benefits to the program. 
In this chapter we will examine the 
fusibility of performance outcomes in the 
context of state-financed, workplace-based 
retraining programs. 

Company and Wbrk-Unit 
Peifonnance 

The evaluation framework for company 
and work-unit performance is 
summarized in Figure 4. Essentially, it 
shows that the financial and market 
performance of a business are measured by 
such factors as shareholder value, 
profitability, return on investment, and 
market share. The company's overall 
pa-formance on this dimension is affected 
by the success of its competitive strategy, 
i.e., its ability to choose the correct business 



goals, and its sutxe^ in attaining them. 
The competitive strategy and business 
goals are themselves interpreted in terms 
of operational structures and objectives 
that are designed to affect specific 
performance measure. 

Financial and Market Performance. 

Investor expectations and creditor 
demands essentially define the financial 
and market performance goals of a 
buiiiness. In publicly held companies, 
investor confidence is reflected in the price 
and volatility of the stock. Banker 
confidence in either publicly traded or 
privately held firms is reflected in the 
interest rates charged on the credit issued 
to the business. Adverse investor or 
creditor sentiment towards the company 
will affect significanUy the ability of the 
company to raise equity in the capital 
markets, and it will affect the cost of debt 
in the debt markets. Such sentiment is 
based on the financial condition and 
performance of the businras and on the 
assessments of investors or creditors of the 
ability of the firm to remain financially 
viable and improve its performance. 
According to Porter, "competition in an 
industry continually works to drive down 
the rate of return on invested capital 
towaid the competitive floor rate of return. 
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or the return that would be earned by the 
economist's 'perfectly competitive' 
industry. This competitive floor, or 'free 
markef return, is approximated by the 
yield on long-term govanment securities 
adjusted upward by the li^ of capital loss. 
Investors will not toterate returns below 
this rate in the loi^ run because of their 
alternative of investing in other industries, 
and films habitually earning less than this 
return will eventudly go out of busing" 
(Porter, 1^). 

Competitive Strategy and Business 
Goals. Porter (1986) identifies five forces 
driving industry competition. These affect 
the strength of competition within the 
industry and ultimately industry 
profitability: 

• Industry competitors. Rivalry 
among competitors often is 
regarded as the primary force, 
sometimes the only force, of 
competition. It is characterized by 
tactics like price competition, more 
intense adverti^ng, new product 
introductions, improved customer 
services, or improvements in 
warranties. 

• Potential entrants. The risk that 
there may be new entrants to an 
industry depeids on two factors, 
barriers to entry and the e)q)ecled 
reaction from busin^ses already in 
the industiy. The barriers may be 
economic, such as advantages 
related to scale, or structural, such 
as government policy. Possible 
reactions from businesses within 
the industry, such as price cuts, 
may create new barriers or may 
make existhig ones more difficult 
to penebate. 



• Substitute Products. Substitute 
products are items that can 
perform the same function as those 
produced by a given industry (e.g., 
shopping bags may be made out of 
plastic or paper). Substitute 
products limit the potential retum 
of an industry by pkdi^ a ceiling 
on the prices that firms in the 
industry can charge and still earn a 
profit. 

• Baiyaininft Powers of Buyers. A 
buyer group may ecet a stirong 
influence on an industry under 
certain drcumsbinces by forcing 
down prices, by demanding higher 
quality or services, and by 
encouraging rivalries among 
(smpetitors who supply the buyer 
group. 

• Bargaining Powers of Suppliers. 
A su{^lier group also may be a 
driving force in an industry. In 
certain instances, supptiers may 
exert power by threatei^g to raise 
prices or by reducing the quality of 
the products that they sell to the 
industry. This nuiy squeeze the 
profit out of the industry receiving 
the supplies, or it may increase the 
risk of substitution of products 
from other indtistries. 

Businesses essentially have foiu- strategic 
options within this competitive 
environment. Specific goals may be 
established, such as market share or sales 
volume, that act as indicators of the 
company's success in choosing and 
implementing the correct strategy. Porter 
(pp, 35-38) identifies two as product 
(Ufferentiation and product cost. 
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• Product differentiation creates a 
perception in the industry that a 
firm's ofiiering is unique. Various 
approaches may be taken to 
di^erentiating a product, such as 
design, technology, customer 
service, or deal^ support In the 
Metcraft, Inc. case study, the 
company was focused in part on 
maintaining a strong reputation in 
short order-to-delivery lead tim^ 
and on-time delivery. 

• Overall cost leadership may be 
achieved by rdentle^ efforts to 
improve operational and 
production efficiency, tight cost 
and overhead controls, and 
avoidance of marginal accounts. In 
the Northwestern Steel and 
Simi^on Lumber cases, both firms 
pursued strategies for cutting 
operational costs and overhead in 
industries where price is the 
primary basis for competition. 
Each faces many competitors 
producing similar products. 

The third stratc^c option is to pursue 
quality leadership. Although Porter 
categorizes quality leadership as an 
example of product differentiation, we 
choose to treat it separately because senior 
executive in virtually every ca» study 
dted quality as an impK)rtant hictor in their 
overall business strategy. In fact, quality 
was considered to be the primary basis for 
competition within the industry. For 
example, the Los Angeles chapter of the 
National Tooling and Machining 
Association received finaiKial assistance 
from the California Employment Training 
Panel to improve the overall quality of 
metal mill and lathe work performed by 
machine operators in the precision metals 



industry in Los Angeles and Orange 
counties. The aim was to make the local 
precision metal industry moreconqietitive 
in both don^tic and world maricets. 

The fourth strategy. Porter's thhd one, is 
focus. Essentially, the business 
concentxateson a particular segment of the 
total industry market, using 
differentiation, cost, and quality strategies 
to attack the target population. For 
instance, Harman International Industries 
decided to pursue the mki- and high-aid 
speaker markets. This required new 
cabinet-making technologies to produce a 
speaker box that woukl be consent with 
the quality of sound that the company 
technically was able to produce. Another 
example is Manth-Brownell, which, 
depending on the product line, competes 
on the basis of quality or price in tightly 
focused markets. A precision metal 
turning company, Manth-Brownell 
performs high-precision, short-production 
job runs for specialized industries such as 
gun manufacturers. It also will do 
low-cost, high-volume manufecturing for 
businesses in certain targeted industries, 
such as cable television operators. 

Operational Structure and Objectives 
and Performance TaiECts. The 

operational structure and objectives of the 
firm and rekted performance targets flow 
directly from its competitive strat^ and 
business goals. Based on an examination 
of applications for all cases funded by 
California's Employment Training Panel 
and Illinois' Prairie State 2000 Authority, 
retraining assistance goes to one of seven 
business functions: customer sal« and 
service; product design and 
manufacturing engineering; management 
information systems; production process, 
including production efficiency, quality 
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control and product cycle; facility 
management and maintenance; material 
management and control; and 
administrative support 

In each of the 24 cases that we examined, 
we were able to determii^ the relationship 
between the organizational function and 
the competitive strategy and business 
goals. We also were able to discern the 
reasons that this particular function was 
the subject of attention. For example, 
Lawrence Box and Basket realized that it 
was not able to meet demand for its 
product because its capacity for 
manufocturing baskets of a certain size was 
inadequate. Following the acquisition of a 
new basket-making machine and the 
training of the operator, the a>mpany was 
able to meet customer demaiKis. Unicadd, 
Inc. realized that it could not maintain its 
customer base for computer-aided design 
and drafting (CAD) services unless it 
shifted to CADAH a software package 
increasingly being used by major 
manufacturer in the United Stat^. 

In another example. United Savings ^nk 
in California determined that although it 
offered a broad variety of potentially 
attractive products, it was not sufficiently 
effective in inter^ting its Asian customers, 
the banlc's market niche, in these products. 
As a consequence, bank officials decided to 
improve their customer sal» and service 
operations by giving salK training to the 
customer service staff. 

In each of the applications or funding 
synopses that we reviewed covering three 
years of operations in California and 
Illinois and one year in New York and 
Missouri, we were able to determine the 
operational objectives that were used to 
justify the training request (e.g., improve 



product quality, increase sales, reduce 
down timi;, etc.). In each of the 24 cases 
that we studied, companies were able to 
describe specific performance m^ures 
associated with the operational objectives. 

For example, Pirelli-Annstrong Tire had 
to achieve a specific savings in production 
efficiency as measured by labor 
productivity (pounds per person) and 
make other changes in the operation of its 
Hanford, California, facility or face 
closure. Northwestern Sted and Wire had 
to reduce its overall maintenance cc^ in 
ord^- to compete with modem Amerk:an 
mini-mills. Iliese maintenance costs were 
measured based on labor costs per ton of 
steel produced. KLM Industries 
implemented a new management 
information system to improve the quality 
and scope of information available to its 
operations manager for the moulding 
department, to reduce its receivables 
account arui to take advantage of discounts 
offered by its vendors. Possible 
performance measures include 
end-of-month balance in the receivables 
account, savings due to discounts on 
puxchases, averagedaily cash on hand,and 
average length of order-to-delivery time 
(the last also may be considered to be a 
measure of customer sales and service 
operations). 

Figure 5 lists various operational 
structures and objectives and possible 
performance mc^ures. In each of the 24 
cases, representatives of the business were 
able to articulate specific performance 
measures that would indicate its succ^ in 
reaching its ob^ctives. In other words, 
each business was able to describe a means 
of quantifying a change in performance 
associated vdth the operational objectives 
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that served as the basis for justifying the 
retraining project. 

In sununary, we found that in each of the 
24 case studies, the project administrator 
for the business as well as senior 
management (if different from the project 
administrator) or the business proprietor 
in very small enterprises was able to 
articulate the competitive strategy and 
business goals of the Bxm, tie then to an 
operational structure and specific 
objectives, and describe sj^cifk measure 
of performance. In every instance, the 
business was able to demonstrate that it 
collected data on these memires aiui used 
these data to monitor performance. Asa 
consequence, we do not believe that it will 
add to the burden carried by companies if 
the state requires that they articulate 
generally their competitive strategic and 
business goals, identify specific 
operational objectives, and describe 
rebted perfbnnance measures already in 
use. 

Although the path that connects 
operational objectives with financial and 
market performance appears to be well 
marked and direct, it is, in foct, treacherous 
to follow and very difficult to establish in 
most cases. Presumably, a company that is 
able to operate by selling high-quality 
products at the lowest possible price 
should do very well in the marketplace. 
Although this appears to be a nect sary 
condition to sustained financial and 
market performance, it is not a sufficient 
condition. 

A company that achieves its operational 
objectives essentially meets the conditions 
that support the implementation of its 
competitive strategy. It has produced a 
prtKiuct of the quality and at the price it 



intended. However, other elements affiect 
the outcome of the strategy, such as the 
maticet demand for the jwoduct and the 
ability of the company to distribute it cost 
effectively. The outcome of the strate©r, in 
turn, is but one element in the financial and 
market performance of the firm. 
Consequently, slates looking for the effects 
of their investments in retraining will have 
to be content with tracking changes in 
specific operational performance 
indicators. 



"Baining Objectiy^ 

The distinguishing feature of customized 
training programs, including 
workplace-based retraining programs, is 
that state funds are used to meet 
company-specific training objectives. As 
described in the previous chapter, training 
objectives refer to the new behavioral skills 
or work tasks that an individual is required 
to perform to meet job performance 
expectations. The evaluation system 
proposes that trainfaig objectives should be 
defined clearly in a project proposal and be 
measured after training in terms of the 
percentageof trainees reaching proficiency 
levels in each training objective. In 
reviewing the feasibility of using training 
objectives as performance outeomes, we 
firat review the types of training and 
testing done by the 24 companies in the 
case studies. 

Types of TiaininR. Although all of the 
employee retraining funded by the four 
state programs in the 24 training projects 
was customized to company needs, most 
training projects involved a substantial 
amount of training in basic skills and 
vocational and occupational skills that are 
applicable to a broad range of jobs in other 
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companies. The type of tzaining in these 
projects is best described as "functional 
context education" (Sticht, 1989) or 
"woik-based learning" (U5. Department 
of Labor, 1989). The training projects were 
all deigned to transfer new knowledge 
and skills within the functional context 
where the learning was to be applied. 
Ahnost all training projects involved a 
training design that integrated classroom 
instruction and hands-on laboratory 
training with struchired on-site training 
where skills could be applied on the job. In 
the classroom component, trainees 
continually were given examples and 
illustrations of new conapts and practices 
based on well-known, current work 
experiences. 

Asshown in Rgure6, the typesof training 
can be classified into basic skills training, 
vocational or occupational skills traming, 
and firm-spedfic skills. Basic skills include 
a wide range of areas identified by the 
American Society of Training and 
Development in their report. Workplace 
Basics:The Skills Emphifers Want. These 
include group communication skills, 
problem-solving skills, team-buikiing, and 
applied math. Some basic math education 
(e.g., trigonometry, algebra, etc.) was 
included in blueprint reading courses. 
Basic skills training also includes computer 
or machine orientation for employees not 
famiUar with modem technology in the 
workplace. Finally, basic skills include 
business education; i.e., training 
employees to understand the competi'ive 
pressures in their industry and the basics 
of business finance and operation so that 
they can better understand the company's 
reasons for the retraining project. 

Basic skills instruction was a component 
ill almost all training projects. Usually, it 



was integrated with occupational training. 
Examples of extensive basic skills training 
include the shop math training conducted 
by Dresser-Rand in New York and Solar 
Turbines hi California. 

The vocational and occupational training 
corresponded very closely to the 
instructional materials used by public 
educational institutions such as 
vocational-technical centers and 
community colleges. Examples of this type 
of occupational training ran be seen in the 
extensive vocational curriculum used by 
New United Motors Manufacttuing, Inc. 
(NUMMI) and General Motors-Wentzville 
in skilled tradra training for maintenance 
employees arei the CNC training provided 
by the Los Angeles chapter of the National 
Tooling and Machining Association. 

The kinds of firm-specific training varied 
greatly in their transferability to other 
companies. The classic example of 
nontransferable skill training is 
standardized work. In standardized work, 
employees are instructed on a sequence of 
specific job tasks that they should carry out 
on ? routine basis. This type of training 
involves Hmiled classroom instruction, 
usually with visual aids and physical 
demorwtrations. &)metimes this type of 
training was integrated systematically 
with continual process improvement and 
safety training. For example, NUMMI 
received a training grant in 1987 from the 
Employment Training Panel to retrain 
assembly workers in standardized work 
and in Kaizen, a widely used Japanese 
approach to continual process 
improvement. The training program also 
included ergonomics and safety fraining. 
Clearly, the specific job tasks learned by 
trainees were unique to NUMMI and the 
assembly process utilized for that product 
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Figure 6 

lypes and Examples of Training In 
State-Flnanced Workplace-Based Training Programs 
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model However, the Kaizen training did 
provide training in general problem- 
solving that quidkly is becomii^ a basic 
and occupational skill requirement for 
assembly workers. 

Training Objectives and Instnictimial 
Design. The model of baining design 
described earlier suggests that businesses 
should first establish clear training or 
behavioral objectives and then develop 
training modules based on the typ^ of 
knowledge or skills required to achieve 
these objectives. Wherever possible, 
knowledge and skill shouM be measured 
against clearly defined competencies. In 
general, the type of training funded 
through state programs is quite consistent 
with this model However, the training 
projects addressed in this study varied 
considerably in their correspondence to the 
model 

There was wide variation in the d^ree to 
which training objectives were formalized 
and written down in conjunction with line 
managers and supervisors and 
subsequently presented to trainees. In 
some training projects, the line managers 
originated the project and established 
specific training objectives before the 
project began. In other projects, the 
training objectives were jointly determined 
by line management and workas with the 
support of training staff or an outside 
training provider. In other cases, the 
training objectives remained ambiguous 
until after the training program was 
underway. In these cases, training 
objectives were established informally 
during the application part of the training 
program, but they couM easily be defined 
by either the line supervisors or the 
training project director. 



In all 24 training projects, businesses, in 
coop^tion with state agency staff, had 
devek>ped a formal training outline listing 
asequenceoftiainingmodules or activities 
and the general tim^es for eacK Tbsae 
outlines varied tremendously in length 
ami amount of de^ Foranallcon^Tanies 
and training pro^cts consisting mostly of 
structured on-site instruction, the outlines 
were mainly a list of topks to be covered 
with each trainee. In kxge companies and 
training projects involi^ a substantial 
amount of classroom and laboratory 
training, the outlines were more detailedl 
indicating both topic modules, training 
content, aiui instructional n^od such as 
structured on-«ite training or dassnmm 
and laboratory training. 

In most cases, the training materials used 
in the state-funded projects were 
competency-based, that is, the sets of 
knowledge and skill competencies 
required to produce the training were 
clearly defined. For the purposes of this 
report, "training Directives" refer to the 
ultimate behavioral objectives of the 
projects, and "skill competendes" refer to 
intermediate training outcomes that mtist 
be combined and built on in order to 
produce the functional skills necessary for 
successful job performance. In other cases, 
these competencies were klentified easily 
by the training manager or the instructors 
but wQ'e not formalized in the training 
materiala 

Trainee Post-Testing. Qearly defined 
and measurable training objectives 
prov^e the basis for criterion-refierenced 
post-tests. These tests are the best 
indicators of the level of pb performance 
that will support the performance 
objectives of a work imit. They also 
provide the basis for businesses to evaluate 
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theserviccis provkied by their own training 
departments or by outside vendors, 
including public educational institutions. 

Businesses conduct post-tests in their 
training programs for a variety of reasons- 
Some use post-testing to evaluate the 
peribnnax\ce of their training dq>artments 
or outside training vendots. Others do 
post-testing because they need to know 
what employees know or can do l^cause 
of overarching quality control and safety 
considerations. Finally, some companies 
do post-testing because of promotion 
requirements or compensation issues in 
knowledge-based compensation systems. 

Companies utilize three basic methods 
for conducting post-tests on training or 
behavioral objectives: (1) ^ perfbrmaiKe 
is observed by trainers or supervisors, (2) 
trainees demonstrate skilk in a kboratory 
or a simulated ^b envbonment, and (3) 
trainees report what they've learned. Job 
performance assessments can be based on 
the observations of standard work tasks 
during the normal routines of work or 
through some type of standardized 
company reporting system based on 
productivity or quality of output Trainee 
self-reports are usually based on some type 
of post-trainhig questionitaire in which 
trainees are asked whether tluy feel that 
they are proficient in a list of functional 
skills or job tasks. 

Few training projects in this study used 
formal post-tests on training or behavioral 
objectives. Some businesses recognized 
the need for some type of post-training 
evaluation but were unsure how to 
introduce formal testing. Other business^ 
felt that formal post-testing would be 
counterproductive. They preferred to get 



trainee reactions and informal feedback 
bom trainers and line manage. 

However, some business&i did formal 
post-tests ona subset of training ol^ectives. 
Some used laboratory exercises to test 
profidexKy in functional skills. Others 
used computer simulations to test 
proficiency on newcomimtersystans (eg., 
MRFtrairing). The most common form of 
post-testing was written tests on basic 
skills (e.g., shop math) and general 
occupational knowledge and skiUs (e.g., 
geometric dimensioning and tolerandng, 
industrial electricity}. 

Trainee Fre-Testing. Most companies 
dkl not conduct formal pre-tests in their 
training projects. The importance of 
pre-testing(Mered significan&y according 
to the type of training project In projects 
involving mostly standardized work and 
structured on-site training, pre-testhig was 
not considered necessary because prior 
skill requirements were minimal and the 
company wanted to make sure that all 
employees agreed on how best to do a 
particular job. In other training projects 
such as quality control and statistical 
pnxxss control, companies saw the need to 
start all trainees with the basics so that the 
group could develop a standard 
knowledge of and commitment to the 
compan/s effort to improve performance, 
in these kinds of training programs, 
companies wanted to acculturate trainees 
(employees) to a common set of attitudes 
andcommitmentsand improve teamwork. 

Pre-testing was most common i*. 
technic*! training that required strong 
basic math and blueprint reading skills. In 
these training projects, most companies 
coiuiucted at least a short, informal test to 
see whether remedial instruction was 
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necessary. The companies that did not do 
pre-tests for this type of training felt that 
unrecognized problems in language and 
basic skills prevented them from reaching 
their learning objectives with some 
trainees. 

Employee TesUnc Problems and Issues. 
Bmployee testing in the workplace raises 
some difficult issues for companies and for 
state programs. Two central issues are: (1 ) 
the validity of the tests in measuring job 
proficiency, and (2) the use of testing in 
making employment ai^ ^b assignment 
decisions. Some businesses and trainees 
expressed concerns about testing in 
training pro^cts. One major concern was 
that some types of tests, such as written 
tests, are not good predictors of whether a 
trainee can apply skills successfully on the 
job. Also, trainer or supervisor 
observations would not necessarily be 
reliable methods of testing the job skills of 
trainees. A related concern was about how 
companies would use t&t r»ults. 

MeasurinE Training Objectives: 
Practical Illustrations. Given the potential 
validity problems of employee testing and 
the mixed reactions of businesses and 
trainees, state programs should probably 
proceed cautiously in using training 
outcome measures in a performance 
assessment system. However, the 
measurement of training outcomes is still 
critical in insuring that companies 
implement training programs designed to 
achieve bottom-line performance results, 
i.e„ the functional skills necessary to 
improve job and work-unit performance. 

One alternative is to require that 
companies indicate through a simple 
checklist whether or not the trainee was 
proficient in each objective included in the 



training plan. This alternative can be 
illustrated by describing the monitoring 
system used by some companies and 
training contractors included in the case 
studies. 

Glendale Community College used a 
competency-based curriculum in its office 
automation programs that specified the 
training objectives for each module of the 
currkulum. As described in the second 
volume of this report, Glendale also had a 
monitoring system that asked the trainer 
and trainee to complete a checklist on the 
training objectives in which the trainee had 
achieved proficiency after training. 
Thrifty Corporation used a similar system 
for checking progress and conducting 
individualized follow-up instruction. 
Other examples for Aniata Graphics and 
Unicadd are provided in the case study 
summaries. 

This type of monitoring can provide a 
useful model for measuring the attainment 
of training objective It provides a simple 
yet reliable indicator of contract 
performance without having the business 
incur sul^tantial additional coste. 



Worker Earnings 



Federal and state employment and 
training programs have traditionally used 
post-program wages and earnings as an 
indicator of success. The assumption is 
that wages and earnings are an unbiased 
indicator of the incr^sed productivity of 
workers undergoing government- 
financed training (Bishop, 1987). As 
indicated previously (Creficos and Sheets 
1989), this assumption is difficult to apply 
in retraining programs where the trainee is 
employed before and after training. In 



48 

30 



Chapter Four 



employee retraining, the relationship 
between higher skills and higher wages 
will be contingent on many factors 
including the type of compensation system 
in the company and the competitive 
pressures of the company in its own 
industry. 

Compensation Systems. The 24 
companies included in the case studies had 
substantially different compensation 
systems. The most common system was 
fixed wage and salary ranges for job 
classifications, modified by some type of 
merit increase system. In Job 
classifications imder collective bargaining 
agreements, wage rangra were relatively 
fixed. There was a formal system of 
internal bidding for higher wage jobs 
within the company. Some companies also 
had profit-sharing or gain-sharhig systems 
that provided workers with additional 
compensation based on company 
performance. Two companies had some 
type of knowledge-based compensation 
system in which workers couJd receive 
higher base salaries if they had the skills 
necessary to j^rform multiple pbs. 

Because of these differeiKes, it is very 
likely tl at the direct effect of training 
interventions on wages will be contingent 
on how the compensation s)^tem rewards 
performance improvement in the short 
run. If pbs have relatively fixed wage 
ranges with only small amounts possible 
for merit increases, then the training effect 
will be relatively small. However, in 
knowledge-based compensation systems/ 
there should be a more direct relationship; 
completion of training should be 
associated with a significant pay increase. 
Gain-sharing and profit-sharing plans will 
make the effects extremely uncertain 



because of all the factors that influence a 
company's profitability. 

Other Determinants of Wages and 
Eamimw. The expected relationship 
between performance and pay that, is 
assumed in employment and training 
programs also is expected to hold in 
employee retraining programs, evciytWng 
else being equal. However, in companies 
undergoing major organizational and 
technological changes as well as employee 
retraining, the entire organization and the 
industry fai which it competes usually are 
undergoing a major transformatioa Si«A 
companies often are under strong 
competitive pressures to substantially 
Induce costs, improve quality, smd shorten 
delivery times imder their current costs. 
Many times, retraining is a strategy to 
retahi employees at current wage levels in 
the face of strong competition. In these 
cases, a training project is successful if it 
maintains current wage levels through 
performance improvement. 

Because compensation systems in 
businesses vary widely, and increased 
skills,companyperformance,and earnings 
are only loosdy related, state programs 
should be very cautious in how they 
interpret post-training wage and earnings 
information. Although state programs 
should monitor and report this 
information, it should not be a major 
performance outcome measure forjudging 
the success or failure of the contract. 

Measuring Earnings.' Possible UI 
Applications. Because of thdr limitations, 
businesses do not use wages and earnings 
as a performance outcome measure to 
assess the effectiveness of their retraining 
investments. As a result, if state programs 
require companies to report this 
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infonxiation, it would represent an added 
cost to the company for which the benefit 
is unclear. 

One way for a state program to track 
earnings information without imposing 
substantial costs on the company or on 
program staff is to use wage record data 
from the unemployment insurance (UI) 
system. Beginning in 1988, all states 
became wage-reporting. Thus, all states 
can use the quarterly wage records filed by 
each company to track the post-txaining 
wages of each trainee. Frevioiis studies 
have concluded that the ccnnprdiensive 
coverage, low cost, and high reliability of 
m wage reporting systems make them 
appropriate for a wkle range of evaluation 
purposes in JTPA, vocational education, 
and welfare reduction pxt>grams (Stevens, 
1989; Baj and Trott,19W). The only added 
burden is that companies would have to 
submit the social seciuily ntmibers of their 
trainees. All four states participating in the 
project already require the filing of sodal 
security numbers. California and lUinois 
already use UI wage records to confirm the 
employment status of trainee 90 days after 
training for performance-based payments. 

As part of the feasibility study, we used 
the three pilot case studies-Hannan, Rohr 
Industries, and Ingersoll Milling and 
Machining-to test the feasibility of using 
m wage records for tracking pre- and 
post-training earnings. The Emplcyment 
Training Panel provided quarterly 
earnings data for all trainees for Harman 
and Rohr Industrie. Illinois provided 
similar information for IngereoU Milling 
and Machining. This infomdation was 
used in the case studies with no major 
problem. 



Siunmaxy and Condusion 

The case studio of 24 training projects in 
the four state programs suggest that 
company and work-unit performance 
outcome indicators can be identiiied easily 
and measui^ in ways that are consistent 
with the methods tl^t Inisinesses use to 
measure their performance. Such 
measures wouki not impose an extra cost 
on businesses because they normally 
collect the same information fbr their own 
internal purpo^. 

These case studies present a more 
complex picture of the measurement of 
tiaiiting ob^ctiv^ The approach to and 
quality of instructional design varied 
tremendously by training project. 
However, it is clear ttmt most companies 
could easily have restructured their 
training curricula in tenns of clearly stated 
ami measurable learning objective This 
requiremoit would entail additional effort 
from most companies. However, it is 
consistent with widely aa:epted standards 
of business training design, axvi it ^ould 
result in more effective training projects. 
Because of validity and reliability 
problems in employee toting, especially 
employer ratings of pb performaiKe, we 
conclude that post-testing requirements 
shouW remain very simple and easy to 
administer. The major purpose of 
post-testing is making sure that compani^ 
maintain a dear commitment to reaching 
training objectivira for the vast majority of 
trainees. 

Although the use of wages and ^mings 
as a major performance outcome imiicator 
has considerable problems, they should 
still be tracked and reported by state 
programs. Data collection costs can be 
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zninimized if wages and earnings do not 
become a central performance indicator 
and if UI eanUi^s records aire tlte major 
source of data. 
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CHAPTERS 

RGETING TRAINING INVESTMENTS: 
Business Screening and Substitution 

Minimization 



The fourth feasibUity question is whether 
it is possible to develop appropriate 
business selection and screening 
guidelines to improve the tai^eting of 
training investments. This chapter 
presents findings on the feasibility of using 
the seven proposed screening guidelines. 
Based on these findings, this chapter then 
presents an application process that could 
be used by state prograna in selecting 
businesses using these screening 
guidelines. 



Business Screening: 
Competitive Strategy, 
Performance Objectives, and 
Need for Retrainii^ 

Based on our review of approved projects 
in California, Illinois, New York, and 
Missouri, we are confident that businesses 
are able to describe their competitive 
environments and strategies, their 
business goals and operational objectives, 
performance measures, and the 
importance of training objectives. 
Inasmuch as a substantial portion of this 
information is included, sometimes 
incidentally, in the narrative of a funding 



request, we do not believe that the 
transaction time or costs h\curred by either 
the funding agency or by the businew 
seeking assistance will be Increased 
appreciably if such information is made 
formally a part of the grant request. 

Evaluating »t»g ApoUcation forFunding. 

In general, we conclude that asking 
companies to supply this type of 
information is both reasonable andfeasible 

in terms of th& transaction tx^\s for the 
company in applying to a state program. 
The following issues remain. 

• To determine whether it is 
appropriate to use this faiformation 
as the basis for selecting companies 
to fund. 

• To determine whether such 
information from the company is 
sufficient to form the basis of a 
funding decision. 

• To determine the nature and 
availability of other hiformation 
that is required for a funding 
decision. 

State-financed, workplace-based 
retraining progranns are based on two 



35 



5f; 



State-Financed Retraining Programs 



principles: (1) giving assistance to 
businesses to help them become more 
competitive (thereby supporting the local 
or state economy), and (2) impxx)ving the 
long-term economic security of the worker 
(first/ by helping the company prosper and 
second, by making it possible for the 
worker to gain marketable skills). In order 
for the state to makea reasoimblejiuigment 
regarding the likelihood of success with 
either goal, it must beawareof a company's 
current condition and the direction that it 
intends to take. Although the training 
component may represent a small portion 
of the total cost of making improvements 
in a business requesttag assistance, the 
reason for the training can be understood 
only in the context of the overall effort. In 
addition, as competition for government 
retraining assistance increases, the state 
will have to make increasingly finer 
distinctions among applicants. This, by 
itself, will add to the demand for 
information on the competitive 
environment and direction of the 
applicant. Therefore, it appears that the 
state has no choice but to require 
businesses that seek assistance to explain 
their circumstances and describe their 
plans. 

Once it has received information about a 
business, the state is obligated to test the 
reasonableness oi the company's 
assumptions and conclusions. In many 
instances, th? business itself may provide 
the documentation that supports its goals 
and operational objectives. Often, this 
documentation is in the form of 
benchmarks. Either these benchmarks are 
generally accepted within the industry or 
they are standards established by a 
business acknowledged to be a leader in 
the characteristic that the applicant wants 
to emulate. For example, Lyphomed 
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provided extensive documentation on its 
interpretation of the "good manufacturing 
practices" rule of the Food and Drug 
Administration as the industry standard. 
In this case, the company was able to show 
howithadaddressedeachconcemdtedby 
the FDA at the conclusion of its last 
inspection and that the news of FDA 
concerns had adverse consequences on the 
financial health of the company. In 
another example, Harman had introduced 
speaker cabinets that, because of their 
unique design, required manufocturing 
tolerances that exceeded industry 
standards for speakers in the same price 
categories. Harman sought to be the 
quality leader in cabinet design and 
manufacturing. Through its quality 
control records system, it documented 
adverse consumer reactions to poorly 
manufactured speakersand the changes in 
consumer opinions as improvements were 
made in n\ill operations. Another 
company, Manth-BrowneU, provided a 
copy of an iidustiy-sponsored research 
report on the precision metals industry that 
formed the basis for company owners to 
target specific niches in the precision metal 
turning business. A fourth company. 
Northwestern Steel and Wire, reported the 
industry-wide standard for maintenance 
costs for American mini-mills and set out a 
strategy to reduce its maintenance costs so 
that it could become more cost-competitive 
within its industry. 

In some instance, a state agency will not 
be able to rely on the applicant to provide 
the benchmark information. This will be 
true in particular for small businesses that 
have few if any local competitors. 
Although many small businesses have a 
very clear idea about the problems that 
they must overcome to remain viable and 
grow, very few have the capacity to 
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document market conditions. In such 
caseS/ the state should be prepared to offer 
technical assistance, either through the 
agency or in cooperation with state 
business assistaiu:e programs, that will 
assist the business in gaining a 
comprehensive view of competitive 
standards and corditions. It is possible 
that the markets served by the enterprise 
are so specialized that no industry 
benchmarks exist. In such circumstances, 
all that the state can reasonably expect is 
that the business explain how its financial 
or market performance will improve with 
the changes it proiK)ses. 

In very laige grante, those in excess of 
$100,0(X}, we believe that it is justified for 
business and government to incur the 
added transaction cost for infom ation on 
relevant benchmarks if the information is 
not provided initially. A variety of readily 
available sources provide information on 
the competitive strategy and business 
goals of a firm. In the case of firms with 
publicly traded stcKks, annual rejK>rts and 
10-K filings with the SEC are available to 
the public. Privately held firms often have 
brochures or other public information 
items that discuss the products and 
services they offer and the reasons for 
doing business with them. A more 
detailed understanding of the competitive 
strategies of either type of business may l« 
obtained easily in interviews of senior 
management and often in interviews with 
the primary contact at the firm. However, 
many business^ may be sensitive about 
revealing such information in public 
documents for fear that it may give 
competitors an early opportunity to plan a 
response to a new strategic initiative. 
Consequently, considerable care must be 
taken to avoid revealing proprietary 
information either by publtehing it or by 



allowing it to become the subject of a 
freedom of information actioxL 

The confidentiality problem is being 
confronted by all state economic 
development programs. States should 
review their freedom-of-informatlon 
statutes and make sure that company 
applications can remain con&lential and 
protected from external inquiries. 

Sources of information on industry 
baidunarks include industry associations, 
market experts, government studies and 
publications, inter-firm productivity 
surveys, news reports, and custon^iB. For 
example, the National Tooling and 
Machining Association, the Precision 
Metal Association, and the American 
Electronics Association puUidi financial 
ratios and performance information for 
different typ^ and sizes of manufacturing 
companies in their industries. Industry 
analysts publish annual reports on the 
performance of North American auto 
assembly plants. In ligltf of the sen^tive 
relationship that exists between supplier 
and buyer, information from customers 
should be obtained only in cooperation 
with the applicant. 

Skill Mismatch and Skill Shortage, The 
next two screening criteria refer to the need 
for retraining: the mismatch l)etween skills 
and job requirements and the demand for 
the new ^dlls in the effective labor market 
(i.e., the labor market area defined by the 
normal mobility of workers within a given 
occupation and industry). In considering 
the question of skill mismatch, the most 
important factor to consider is whether 
companies can define any new and 
measurable training objectives that are 
linked to oi^nizational and technological 
changes ai^i the resulting redesign of jobs. 
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In Chapter 2, we propc»ed tluit businesses 
b« asked to define dear and measurable 
training objectives that list the new 
behavioral skills needed to meet the new 
job requirements. The next factor is the 
degree of mismatch created relative to the 
size of the state grant requested. This 
determination largely should be the 
judgment of program staff and should not 
require additional outside coiihultation or 
research. In most of the 21 cases, 
businrases were able todefinea widerange 
of new skill requirements and were able to 
show how they created a sigr.iHcant 
mismatch. This requirement ^uld not 
pose a major problem for businesses 
applying to state programs. 

The second issue of marketable skills is 
more difficult to address. In most cases, 
businesffs could comment on the denuind 
for the new skills in their immediate lalx)r 
market area. They were able to explain the 
difficulty they had in firming employe 
with these skills and the common problems 
of businesses in their industry to recruit 
similar people. In cases where a training 
provider served a consortium of small 
businesses, skill shortages were quite 
evident and easy to address. The Los 
Angeles chapter of the National Tooling 
and Machining Association created its 
CNC training program to fill skill 
shortages of machinists and machine 
operators. 

Although most companies can address 
the skill shortage issue, the state program 
should do additional work to address it, 
specially when the grant is justified on 
that basis. In the screening guidelines, we 
propose that a business be able to justify 
the need for retraining assistance on the 
basis of improvements in business 
performance and training in marketable 



skills. The latter insures that, if trainees 
lose their jobs, they will most likely be able 
to find employment in their immediate 
labor market area. If potential 
improvements in business performance 
are difficult to assess in the application 
prcx:e^, then state program stafif should 
put grater emphasis on marketable skills 
in making their final decision on the 
training project. 

^te program staff should establish the 
followtog minimal criteria for determining 
marketability. First, the occupation or 
occupational chister for tiie skill training 
(e.g., machinist) should not be declining 
sigiU&antly in the state. Second, industry 
representatives outside the company 
should confirm that the new skiUs are in 
demand by similar companies in the labor 
market area. The first condition could be 
established through state or local 
occupational projections developed by the 
Occupational Information Coordinating 
Committee of the state. Because skill 
demand and occupational demand are not 
equivalent, great caution should be used in 
drawing any conclusion regarding 
marketability unless the occupational 
cluster is tmdergoing drastic employment 
decline. 

In meeting the second condition, state 
program staff could contact a 
representative from an industry 
ass(x:iation axvi get the names of at least 
three personnel directors who could 
comment on the skill requirements for that 
occupation in that industry group. The 
personnel directors could confirm that the 
training is for skills that are in demand in 
the labor market area. Because state 
programs do lecuning types of training 
(e.g. SPC, MRP, CNC), the survey of 
personnel directors would not have to be 
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repeated for every contract. In addition^ 
state staff would not have to ask for 
detailed information from personnel 
directors. They would require only 
enough information to establish that there 
is at least some market demand. 

Substitution Guidelines: 
Need for State Assistance 

Although the state may be satisfied that 
the training is tied to legitimate and 
sensible business g(»ls and mrasurable 
operation objectives, the question remains 
whether state assistance is required to 
bring about an appropriate training 
program in a timely fashion. We have 
conditded from our case studies that the 
issue of substitution can not be und^tood 
properly as a question of whether the 
company has sufficient money to jm.y for 
the training without outside assistaiKe. 
Although that should be one 
consideration, a comprehensive set of 
substitution guidelines should r^ognize 
four major barriers that could prevent a 
company from retraining its workers at the 
level and in the time needed to accomplish 
its performance goals. These barriers are: 
(1) financial need, (2) the absence of prior 
training programs, (3) marketable skill 
training, and (4) layoff decisions and 
possible corporate disinvestment. 

Financial Need. The Prairie State 2000 
Authority addresses the issue of financial 
need directly by requiring the applicant to 
show that it lacks the financial resources to 
do the training. Businesses considered 
eligible for state assistance are those 
Incurring operating losses or reinvesting 
retained earnings without paying 
extraordinary officer salaries or dividends. 



Also, money-losing facilities of otherwise 
profitable parent corporations are 
considered eligible for Prairie State 2000 
as^stanc^. Bnally,bu^es5e8 that already 
have undertaken the retraining project 
prior to funding approval are 
automatically considered ineligible for 
assistance. The Board of the Authority has 
concluded that by commencing training, 
the business has demonstrated a capacity 
to pay for it. 

PriorTraininft Experience. The approach 
of the Prairie State 2000 Authority is based 
on the assumption that the chief barrier to 
training is finandal. Although company 
officials in all 24 C3S6S reported that the 
state funds were responsible for the 
decision to undertake a comprehensive 
training program in a conc^trated time 
paiod, they also reported that assistance 
fit>m the state helped to overcome other 
barriers related to the lack of previous 
experience with and commitment to 
employee training. One such barrier often 
dted was that senior management was not 
sensitized to human resources problems 
created by technological or process 
changes. It app&sred that state assistance 
made it possible to focus the attention of 
senior management on the need to train. 
For example, Rohr Industries had failed in 
a previous attempt to implement a 
rudimentary version of the manufacturing 
control system that it subsequently 
adopted. One reason for its initial failure 
was that management thought it was 
sufficient to train only key personnel. The 
prospect of state assistance was reported to 
help in the decision to train all personnel 
on the sub^uent system. 

A related barrier was the reluctance of key 
managers to disrupt company operations 
for training. However, it appears that the 
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prospect of state involvement and its 
continued piesence following 
authorization of the grant gave Rohr the 
impetus it required to take people out of 
productive work and sei^i them to class. 
Although production managers may 
support the id^ of training, many were 
reported to be skeptical about its net 
benefits in light of the problems that it 
would create in meeting daily production 
objectives. Assistance from the state 
appeared to give management an 
important incentive to work out these 
problems between production managers 
and those responsible for training. 

Another barrier appeara to be fear of the 
unknown. Companies that never had 
undertaken a formal training program 
(except for new employee orientation or 
safety training) reported that the prospect 
of state assistance helped to convince 
management to try it on a ki^ scale. Asa 
consequence of Uieir experience, some 
companies have established on-going 
training programs. Solar Tuibines did not 
offer fbnnal skill training to its machinists 
until it received a grant from the 
Employment Training F^nel. It since has 
opened a training department that serv^ 
its own workers and provide training 
services to other southern California 
businesses. 

Training in Marketable Skills. One 

policy rationale for state assistance is that 
it compensates companies for the 
externalities of employee retraining. The 
assumption is that busine^e will not 
retrain employees to the full extent 
nece^ry unless government compensates 
them for the risk of investing in workers 
whom they may lose to other companies 
before they receive the full return on their 
investment. The risk is that competitors 
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will be able to bid away skilled workers 
without bearing the direct cost of training. 

Although the research literature gives 
great importance to \he problem of mOning 
externalities, this barrier was only one of 
many reported by companies. It was most 
often reported by small companies and 
companies retraining workers in 
occupational areas of strong business 
demand. 

As described above, it is very difficut for 
a state agency to determine market 
den^ind for skills on the basis of available 
secondary labor market information. 
However, this is still an important 
screening guideline for minimizing the 
effects of substitution. 

Layoff Decisions and Corporate 
Disinvestment A final consideration was 
whether the business establishment 
applying for state assistance was 
attempting to improve internal operations 
in the context of some type of corporate 
investment decision in the facility. A 
related issue is whether the business 
establishment was actively considering a 
layoff decision. 

In some cases this consideration was 
importa nt in determiiUng the need for state 
assistance. For example, the Employment 
Training Panel provided funding 
assistance to Pirelli-Armstrong Tire 
Company after being notified that the 
facility might close. State retraining was 
used to help ihe fedlity remain open. This 
example illustrates how such screening 
criteria may be useful in approving 
training requests. 

In our 24 cases, we found that available 
financial resources were one of several 
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factors that contributed to the reluctance of 
the company to initiate retraining. The 
operational difficulties in scheduling 
training, poor experiences with previous 
ill-coiKeived training efforte, aiKi fear that 
the txaining would not succeed vf&re a few 
examples of the barriers that prevented 
employers from initiating their own 
training programs. In these cases, the state 
grant validated the legitimacy of training 
and overcame both financial and 
management uncertainties. 

Implicatioiis for State Policies 
on Matching Coittributioiis 

The four states participating in our study 
each recognize the contribution towards 
the training, either direct or in-kind, that is 
made by or in behalf of the company. In 
Illinois, the amount of the state 
contribution is tied to the percentage of the 
direct training costs. The California 
Employment Training Panel is required 
under its new legislation to ^tabli^ a 
policy on in-kind contributions with 
respect to its funding decisions. New York 
and Missouri variously consider the 
relative shares that each party must bear in 
the training program. 

Generally, states require in-kind 
contributions for any of three reasons. One 
reason is that the state has limited financial 
r^ources and uses formulas for in-kind 
contributions as the basis for extending the 
reach of its programs. Another reason is 
that the state uses the in-kind contribution 
as evidence of a company's commitment to 
the retraining effort. In using this reason, 
the state pr^umaWy is able to show that 
the company views the retraining as 
sufficiently important to require the 



expenditure of its own resources. The state 
also hopes that if the company has a stake 
in the training, it will exeidse prudence in 
the costs that it incurs in the training itself. 
Finally, a state may use in-kind 
requiren^ts as a means of controlling for 
substitution. This prindpally is based on 
the assumption that a company will not 
incur stibstantial costs in onler to earn a 
small increment of revenues from the state 
coming in the form of training as^stance. 
However, this use of in-kind policies 
appears to be done ad hoc. 

In-kind policies may be adjusted and 
formalized to address substitution issues. 
We propose that the funding burden 
carried by the state be inversely related to 
the likelihood that the company will do the 
training anyway without assistance. 
Based on our interviews^ upper 
management and production managers 
regarded classroom training as least Hkely 
to be done without assistence. From a cost 
standpoint, classroom training requires 
that the company incur the cc^t of the 
trainer as well as the \c»t production time 
of workers who receive their r^ular pay 
but are away from the normal routines of 
work. In addition, compani^ appeared to 
have the least experience with classroom 
training, probably because of its cost. 
Finally, classroom training tends to focus 
on more basic and occupational skills. 
Such skills tend to be more transferable 
because of the many other skills that they 
support. All these factors appear to 
impose significant barriers to classroom 
training. 

At the other end of the training 
continuiun, structured, on-site training 
was regarded with the least resistance by 
upper management and production 
managers. Typically, such training can 
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occur during the course of nonnal work 
routines when some production output 
may result. In addition, companies 
typically had considerable experience with 
the form of on-the-job training in which 
more experienced workers show new 
workera how to operate specific machinery 
or perform specific procedures . Finally, 
ructured, on-site training tends to focus 
.11 job-specific skills. These skills often are 
unique to a particular company and are 
less transferable to other firms, even those 
in the same industry. Consequently, 
businesses were much more likely to 
undertake structured, on-site training 
without much uiging from the state. 

Laboratory training appears to be 
somewhere in between classroom training 
and structured, on-site training in terms of 
the propensity of a company to undertake 
such training on its own. Although it 
carries the same costs as classroom training 
because it takes workers away from the 
normal routines of work, laboratory 
training tends to be less transferable than 
classroom training and seems to more 
common than classroom training. 

In light of these observable differences, 
we propose that states require in-kind 
con* ib tion in accordance with the 
principles diagrammed in Figure 7. 
Essentially, the share of the state 
contribution should be large relative to the 
company contribution when the costs are 
related to classroom training. This share 
should be reduced for bboratory training 
and should virtually disappear for 
structured, on-site training. ITie relative 
shares also may be adjusted to reflect other 
issues as well. For instance, the line 
demarcating state and company shares 
may be moved to the left and made more 
vertical, thereby increasing the state share. 
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if the state determines that the company 
lacks the financial resources to pay for the 
training. On the other hand, the line may 
be moved to the right aiKl made more 
horizontal if the company previously has 
paid for training projects similar to the one 
for which it is seeking stete assignee and 
if it is judged to have s%nificant resources 
for training. This may occur in cases where 
the company is seeking additional funds 
for the continuation or expansion of an 
existing classroom training program. In 
addition, state policy, such as the Illinois 
statutory requirement that the Prairie State 
2000 Authority cover no more than 50 
percent of the direct training costs, vdll 
determine where the Une is placed and 
even its shape. (In the case of Illinois, the 
line would look more like a semi-dxde 
with the state rovering half of the costs of 
classroom and laboratory training and 
nothing else.) R^idless of the specific 
polki^ i ;^t will iavor one type of training 
investment over another, state in-kind 
contribution policies may be structured 
formally to help minimize substitution 
concerns. 

Proposed Application Process 
Using Screening Guidelines 

Based on these findings from the 
feasibility study, we conclude that 
screening guidelines can be applied 
effectively in a simple application process. 
The major components of the application 
are shown in Figure 8 and are discussed 
below. 

1. Overview of the Competitive 
Problems Faced by the Business. In the 
case of publicly held companies, 
competitive problems often are stated 
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Figure? 

Proposed Model for State In-Kind or Matching Contribution Policies 
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Business Application and Screening Guidelines 
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generally inannual reportsand 10-K filings 
with the Securities and Exchange 
Commission C5EC). In closely held 
connpanies and at the plant or functional 
unit level of publicly held companies/ 
details of thi^ competitive problems may 
be proprietary. 

Regardless of the source of information, it 
is sufficient to state the problem generally 
in order to give a context for the strategic 
and operational objectives of the company. 
This will not reveal any proprietary 
information and will not add to the 
competitive disadvantage of the business 
making application for state assistaiure. 

2. Strategic and Operatiimal Objectives 
for Resolving the Problems of 
Competitive Disadvantage. The financial 
and market performance of a business^ as 
defined by shareholder value, profitability, 
return on investment and market share, is 
affected by the ability of the firm to pick the 
correct competitive strategy and to 
implement it effectively. The o^mpetitive 
strategy and business objectives of a firm 
revolve around the factors that descr&e the 
competitive environment for that firm. 
Typically, these factors are one of tl ree 
types: product differentiation, prcKiuct 
quality, and product cost. The operational 
objectives define the fuiKtional imits that 
a^Kt the strategic outcomes of the firm 
and set the level of performance for these 
units. Functional imits include customer 
sales and service, product design and 
manufacturing engineering, management 
information systems, production 
processes, facility management and 
maintenance, material management aiK^ 
control, and administrative supjwrt. The 
success of the company in meeting these 
objectives may be determined on the basis 
of changes in associated performance 



indicators. For instance, quality control 
may be considered to be a problem in 
production proc^es. This problem may 
be manifested by an unacceptable cost for 
waste, which affects the ov«all cost of the 
product. The operational objective wouM 
be to improve the quality of the production 
process as measured by a reduction in the 
rate of waste. We believe that companies 
recognize the need to take action when 
performance falls below acceptable levels, 
and they shoukl be able to articulate their 
performance targets in making their 
request for assistance from the state. 
Inasmuch as the nominal performance 
values often are closely held, states should 
expect to receive performance targets in 
terms of percentage changes. 

3. Company Action Plan and Training 
Objectives. In its efforts to achieve 
operational objectives, a business will 
undertake a series of chan^, including 
the acquisition of new eqtiipment, an 
adjustment in work processes and 
procedures, and changes in productdesign 
and quality standards. In a properly 
executed strategy to achieve operational 
objectives, these actions, including the 
training, will come together in an 
organized and logit^l fashion. An action 
plan is a written description of the actions 
to be taken, the order for taking these 
actions and the associated timetable. The 
action plan shoukl include the training 
pLn and the timetable for other related 
activities (e.g., installing new machinery, 
establishing manufacturing cells) that will 
be combined with training to produce the 
expected change in performance. It is 
especially critical for workers to receive 
their traiiUng at a time that will enable 
them to put their new skills to use. 
Consequentiy, it is appropriate for the state 
to know the action plan so that it can 
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monitor the progress of the company and 
make adjustments in the training schedule 
in order to keep it on track with the overall 
strategy for improving operation. The 
action plan also will define the skills that 
the workers must have and the training 
objectives they must achieve to support the 
implementation of the changes. 

4. Training Design, The retraining 
project that is proposed for state assistance 
shnukl artkulate the behavioral or training 
objectives and the related skill 
competencies that are needed m order for 
the workers to contribute to the company's 
performance targets. The training 
curriculum then brings each worker 
forward to an acceptable level of skill from 
one that is either assumed or determined 
by pre-testing. 

5. Need for State Assistance. In our first 
report^ we suggested that stat^ should 
avoid making investment in firms that 
appeared to be able to afford the training 
without causing any appreciable delay in 
timing or without limiting its scope. We 
believe now that this represents too strict a 
standard in distinguishing needy 
companies from those that view 
government grant assistamre as another 
source of revenue. As a consequence of our 
feasibility study, we recommend that, as a 
condition for the business to receive funds, 
state officials must be satisfied that the 
business faces barriers to initiating training 
and that state participation is necessary for 
the appropriate training to occur in a 
timely fashion. A company that proves 
financial need is, in effect, giving prima 
facie evidence that state assistance is 
required. However, three other barriers 
should be considered: (1) prior training 
experience, (2) training in marketable 



skills, and (3) pending layoff or possible 
corporate disinvestment. 

6. Project Monitoring and Evaluation 
Flan. The applkation should set forth the 
basis for verifying the participation of 
company |»isonnel in the training project 
and for monitoring the progress of 
training, the measures for assessing the 
attainment of training objectives, and the 
performance outcome measures for 
assessing the level of the company's 
training-related succ^ in achieving its 
strategic and operational objectives. In 
setting performance outcome measures, it 
is important to recognize that changes in 
performance may follow several montl^ 
after the completion of training until all 
associated organizational and 
technological changes have been 
implemented fully. 

Summaxy and Conclusions: 
niustiation of Program 
Application 

The feasibility of requiring program 
applications to include the various kimis of 
information discussed in this chapter can 
best be demonstrated by describing what a 
pOMible company application could look 
like for the four state programs in the 
study. 

As shown in Figure 9, the description of 
competitive strategy and business goals as 
well as the statement of operational 
performance objective, business action^ 
and the role of retraining objectives can be 
contained easily within two pages of an 
application. 
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Figure 9 

Example Application for XYZ Corporation 



XYZ Cbipoiation is a medium-sized* fimii>K>wned and cyeraicd business making widgets. It sells these widgets to automotive 
ssseoiUy faeUlties in time midwestem ^tes, where it competes with four otter wkSget manuractuicrs. 



The automotNe imiusny uml to zequire widget vendors lo compete on the basis of price. Recently, vend ore have had to meet 
stringent quality and delivexy requirements while facing demands to tower prfee by 10 pexccnt. XVZ Coiporation faces loss of its 
contracts S it cannot meet oistomer demands within one year. 



XYZ Coiporation is determined to improve quality, reduce throughput time, and reduce price to meet customer demands within 
one year. Sp^ficaiiy, it expects to improve quality as indicated by a IS percent reduction in scrap o^ts by implementing a quality 
assurance program. It abo expects to reduce throu^put time as indicated by a 10 percent reduction in the time it takes to produce 
a widget frmi receipt of raw material to dtipment of final product. Thb wiU be achieved by the implementation of a quality 
assuran^ pr^ram that will reduce re-work costs by 35 percent, h b e^imated that a 35 percent reductk>n in re^ric will reduce 
avera^ throughput time by 10 percent. Finally^ it expects that price will be reduced by 10 percent as a consequence of the savings 
Bchie^ in the quality assurance program, which consists of the imptementation vf statistical process control The company will: 

^ Implement SPC within the next nmc months through a consulting contr^t with the Widget Manufacturing Association. 
O Z>eveIop raw material specillcatiCHts within next six months and implement raw material inspection program using statistical 

sampling techniques. Identify material defects and develop strat^es with suppliers to improve tncir product. 
O Develop and implement a material utilization program within the next six months. 
O Complete tedincdogy and industrial engineering assessments of manufacturing faciiity within next 3 months. 



Figure 9 continues 
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Figure 9 (continued) 
Example Application for XYZ Corporation 



StattelicaJ proccs conlioi i$ entiitly new to XYZ Coiporatton. CcmsequenUy, the entire pioductlon w^orcc amS qwallty 
assujraoce departmenl xequirea training in llie aul^jeci. 



(Curriculum and budget) 



Althou^ XYZ Coiportllon pcriodioiUy hw provided ualning lo its cmplj^M c» pn^r «fety procedures in ctfflfoniwiKe wth 
OSHA, it has never undertaken any broad-bised lefihnlcal training pro^ Management recognizes thai the mo« effective means 
to imptement siailalcal ptocess contnd in the plant is to train aU pioductlon perswuiel However, if the company were to iindertalte 
the project on its own, it would incur liic direct costs of the training program and the loiscs that are the result of lost production due 
lo the number of workers that will be in ctoMroom training and out of the normal loulines «rf wodt. In light erf the risk that this 
training project may not be sufTjcient to achieve the expected results, the company is unwilling to proceed on its own. The assistance 
provided by the siote spreads the risk and brings in a janner thai has had ejcperience in finanring such training projects. 



The XYZ Qjrporaiion expects that training wUl bc^n «dihin two months after rccehdng approval trf the grant. It also expects that 
witWn five months fottowing the commencement of training, CSi percent of the workforce inwrfvcd in training wiU have completed the 
STC project cxereise. An additional 10 pcrecnl of the workforce will complete the training each month thereafter. One month 
following the completion of training, the XYZ Corporation expects to have achieved over the previous year a 10 percent reduction m 
scrap costs per unit of final product, a 10 percent reduction in throughput time, and a 35 percent reduction in the costs of re-work 
per unit of final product. 
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CHAPTER 6 

TRAINING IMPLEMENTATION AND STATE 
MONITORING: What Makes A Training 

Project Work? 



The 24 case studies provided important 
insights into what makes state-financed 
training projects work. In every case 
study, we highlight the different strategies 
(e.g., joint management-labor committees, 
in-house versus contracted training) and 
instructional approaches (e.g., 
computer-based instruction) that 
companies use in retraining workers (see 
Volume II). In virtually all cases^ these 
strategiet appeared to work very well. We 
also reviewed the monitoring activities of 
the state program from both the program 
and business perspectives, especially in 
Illinois and California. In this chapter, we 
summarize findings that could be applied 
to all 24 case studies and the four state 
programs. 

Key Factors in Training 
Success 



The training design model presented in 
Chapter 3 provides useful guidelines for 
state programs to follow in establishing 
business screening guidelines and 
constructing a basis for project monitoring 
and performance outcome assessment. In 
general, we found this model to be 
applicable to most cases and useful in 



highlighting the important features of a 
successful training project. 

Upper Management Commitment and 
Business Objectives. The most important 
factor in determining the success of a 
training project is that it have strong upper 
management support and be organized 
and justified internally to the company 
around a specific and measurable busing 
objective. As recognized by Brinkerhoff 
(1989), training departments in large 
companies sometimes can establish 
training projects that do not meet the 
immediate performance needs of line 
management but rather serve internal 
departmental inter^ts. Small companies 
many tim^ attempt to utilize standard 
training curricula from outside vendors 
without taking the time to customize the 
training package to meet their unique 
performance objectives. In both casK, 
training projects were successful to the 
extent that people learned what they were 
taught. However, they sometimes fell 
short of producing expected 
improvements in business perfonnance. 
The most successful training projects were 
those originated by upper management, 
especially line management, in the face of 
a business problem that required 
retraining. This linkage to a business 
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objective established the basis for an 
effective training design and for the 
cooperation of line managers who had to 
release employees for training. 

Employee Understanding of Business 
and Training Objectives. The training 
model described in Chapter 3 emphasizes 
the importance of providing the trainee 
with a comprehensive knowledge of the 
reasons for training and specific training 
ob^ves. This factor was critical in the 24 
training projects. The most successful 
training programs were those that devoted 
considerable time to involving employees 
in a comprehensivediscussion of the major 
competitive problems faced by the 
company, the specific strategies proposed 
by the company, and the specific 
operational objectives requfaed to remain 
competitive. Succrasful programs also 
ensured that employees understood why 
these operational objectives required 
changes in internal operations and in the 
basic skills required for their jobs. In some 
situations, it was important that labor and 
management develop training programs 
through joint committees. In other 
situations, it was critical that management 
provide extensive information on the 
condition of the company and ask for 
employee input on the design of the 
training program. In still other situations, 
a brief review of business reasons for 
retraining was sufficient to make the 
trafaiing program succ^fiil. The common 
element in all of these training projects was 
that employees fully understood the 
business and training objectives of the 
project and how it could potentially benefit 
them. 

Training Objectives and Com- 
petencv-Based Instruction. The most 
successful training projects were those that 



defined behavioral objectives for the 
training pro^t clearly, based on what 
management and woricers felt were the 
new skills required in the company. Asa 
result, such projects were able to explain to 
the workers the company's reasons for 
trainingand what was going to be expected 
of them in applying their training to their 
jobs. In some cases^ projects used curricula 
that covered the approprkte subjects but 
failed to establish in the thinning how the 
trainees were to apply the knowledge and 
skills on their jobs. This was usually 
worked out while training was underway 
or immediately after the classroom or 
laboratory training was complete and 
structtued on-site instruction had begun. 
In addition, the most successful projects 
used competency-based curricula that 
defined learning outromes dearly for the 
intermediate knowledge and skills that 
were the foundation for the behavioral 
objectives. 

Work-Based Learning. The training 
design model emphasizes the importance 
of using the methods, media, and 
conditions that are most appropriate for 
eliciting the appropriate functional skills in 
on-the-job applications. The most 
successful pro^te were those that linked 
classroom instruction with hands-on 
applications in laboratory sessions or 
structured on-the-job training. This 
work-based learning seemed effective 
particularly with employees in 
nonmanagerial and nonprofessional 
occupations and with those who had 
limited educational backgrounds and 
limited experience with company training. 

Basic Skills Assessment. The training 
design model emphasizes that individual 
differences be assessed before employees 
start the training program. As discussed 
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earlier, this practice was not appropriate or 
necessary in all projects. However, in 
those projects involving technical skill 
training, the most successful projects used 
simple pre-tests for identifying basic skills 
deficiencies. In other cases, training 
promts ran into problems because the 
workers were found to lack important 
basic skills reqmred for advanced technical 
training. 

Government Monitoring: 
The Fcispeclive of Sfcite 
Programs and Business 

State program administrators, agency 
governing boards, state legislatures, 
governors, and businesses hoW the general 
principle that state-financed, 
workplace-based retraining programs 
should be highly flexible and operationally 
streamlined. The proposed evaluation 
system is designed to match the state's 
interests with good business practices. 
Such a design minimizes the possibility 
that businesses will incur additional 
overhead costs that bring little value to 
thek operation, but it provides the state 
with the essential facts about its trainmg 
investment. Nevertheless, it is hievitable 
that conflicts will arise between the various 
interest groups, particularly between the 
slate program staff and the business, over 
monitoring the progress of the training 
project. 

Although state agencies need to avoid 
becoming overly intrusive, it appears that 
the monitoring process frequently 
contributes to the success of the project in 
two ways. 



1. Commitmen t to Training. The 
pressure of daily production quotas or 
customerdemands, the regular assortment 
of business emergencies, and the 
skepticism of workers and managers 
towaids the training project are among the 
many factors that undermine the 
commitment of the business to the 
retraining project It is understondably 
difficult for a manager to take a long-term 
view on the value of training when 
immediate problems must be solved. As a 
consequence, it is common for businesses 
to excuse workers from trailing sessions in 
order to keep them working or to let the 
timetable for a training program slip. 
However, many business project 
managers reported that, because the state 
took an active role in monitoring their 
projects, they were able to enforce the 
company's initial comn^tment to trailing. 
Although many managers said that the 
state's reporting requirements were 
difficult to fulfill they provided a rational 
basis for managers to demand that 
supervisors meet the conditions of the 
grant. For instance, California often 
requires that aU anployees verify that they 
received training by initialing an 
attendance sheet for each classroom 
session- As a result, one company was 
scrupulous about sending its workers to 
training sessions at their appointed times. 
In retrospect, the pro^ manager and 
other supervisors said that the uniform 
participation of all workers in the training 
project was an essential ingredient to its 
ultimate success. 

2. Modif itations to the Traini ng Design. 

In our review of 24casesand in discussions 
with program staff from tiie four states 
participating in the study, we found that 
training designs frequentiy were adjusted 
to reflect chan^ng conditions within the 
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finn. In some cases, these changes were the 
result of competitive pressures causing 
alterations in the operational objectives 
and tai^eted performance levels of the 
company. In other cases, the changes were 
a consequeivce of the training project itself. 
Many businesses reported that in 
implementing the training project, they 
discovered other problems that first 
needed to be resolved before the desired 
business objectives could be reached. For 
instance, it was not imusual for companies 
implementing statistical process control to 
discover that their workers lacked 
fundamental math skills and that the 
training program had to be adjusted to 
include shop math courses. Some 
businesses discovered that by involving 
workers in the design of the training 
program, they were able to understand 
other hidden problems. One company 
discovered that its machinists worked 
from product engineering designs that 
were diffkult to produce. As a result, it 
began to train some nuichinists as d^ign 
engineers to work later in the design 
department. Regular monitoring of the 
progr^ of a training grant gives the state 
early warning about impending changes in 
the grant. 

Although the proposed evaluation 
system stresses the common interests 
shared by government and business, each 
party regards the other with a good deal of 
distrust. Clearly, government staff must 
be diligent in preventing the improper use 
of public monies, and business cannot 
accept unlimited government intrusion in 
its affairs. Nevertheless, significant 
benefits to both government and business 
come as a consequence of their forced 
interaction as training occurs. R^ular and 
systematic monitoring of progress on 
training provides for such interaction. 



Essentially, progress monitoring is the 
process of verifying that the company has 
achieved certain objectives at specific 
milestones. It begins with the application 
when the company is asked to artkulate its 
action plaa At this point the company 
must describe the dements of its overall 
strategy for change and the timing of each 
demoit. The aj^lkation also describes the 
curriculum for the training program and 
the specific training objectives for each 
type of worker. The combination of the 
action plan, the curriculum, and the 
statement of trainiz^ objectives provides 
the basis for future progress monitoring. 

We propose that at the b^inning of the 
training project, the company and the state 
agency establish a schedule that lists key 
milestones and the associated dates. Th^ 
milestones should establish the number of 
people achieving specific training 
objectives. They also should establish 
other key events that are necessary for the 
miccessfiil achievemoit of the company's 
overall operational objectives, such as ihe 
acquisition and installation of key 
production eqidpment. The process of 
monitoring then becomes a matter of the 
state verifying that th^ milestones were 
achieved at the specified time. In the event 
that the project gets off schedule, the state 
and thecompany could use such milestone 
checks to make adjustments in future 
milestones, the funding stream, and even 
the foiKiing amount. 

Some states, especially California, pay 
very dose attention to the amount of time 
that workers aresaid to be in training. They 
require the trainer and the trainee to jointly 
initial time cards each time the worker is in 
a classroom, laboratory, or structured 
on-site training session. We recognize that 
this type of monitoring may be audited. 
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and it has proven to be a useful tool in 
verifying that the state is receiving the 
servkes it paki for. However, by checking 
only on the "seat time" the worlwr spends 
in training, the agency misses the 
opportunity to verify that the worker has 
learned the skills defined in the training 
objectives. 

We believe that the use of milestones, 
including an assessment of the skills 
learned in training by certain target dates/ 
is an effective means of determining 
whether the final training objectives will be 
attained. Such an approach makes 
follow-up evaluation easi^ because the 
essential issue as to whether the learning 
objectives have been achieved is settled as 
the training project is implemoited. In the 
event that the busing is likely to fall 
critically short of its performance targets, 
the state may provide other resources to 
assist it or begin to prepare for the 
possibility that the workers receiving the 
training will need to look for other work. 

Monitoring Perfnnnance 
Outcomes: Data CoUectioiv 
Reporting^ and 
Perfoimance'Based Payments 

The final element of the selection and 
evaluation system 11 the follow-up 
evaluation of the training investment. 
States such as Illinois and California 
require that retrained workers remain 
employed by the company for no fewer 
than 90 days before the company is finally 
reimbursed for the costs of retraining. One 
purpose for this requirement is to 
discourage companies from using state 
assis^nce to train many workers in order 



to select and retain the better on^ It also 
appears to indicate the company's 
commitment to improving its human 
resources. Consequently, we recomuffind 
that Illinois and California retain this 
requiren^t and that it be incorporated in 
the grants to companies in otter states. 

Although states may be reluctant to add 
other conditions to the final payments on 
retraining grants, we x^comoKnd that they 
consider requiring each company to verify 
that the workers receiving retraining 
achieved tiie training objectives. The form 
of the verification nnay vary by type and 
size of bu^ess and by type of training. At 
a minimum, it should represent the 
olservations of the trainer or sups'visor 
that each worker can perform an 
acceptable number of key functional tasks 
identified in the training plan. 

In as much as training is but one factor in 
determining whether performance targets 
are attained, it is not as yet practical to tie 
final grant payments with company or 
work-unit performance. However, we 
suggest that each compan/s pofom^nce 
be monitored, depending on available state 
resources, for the purpose of providing 
program administrators with important 
feedback on the apparent efficiency of the 
retraining program and on the factois that 
may aff^rt the ability of the company to 
achieve its operational and strategic 
objectives. This information may be used 
to improve the quality of the screening 
process and may lead to the new 
performance targets that may reasonably 
be used in routine project evaluations. It is 
important to rea>gnize that change in 
performance may occur up to one yeir 
after the completion of Gaining tmtil all 
associated technological and process 
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changes have been implemented fully by 
the company. 

If states wish to tie performance 
payments to some type of indicator of 
performance improvement, we suggest 
that states focus on training objectives and 
the implementation of the action plan. 
States could make final performance-based 



payments if companies achieved their 
training objectives and completed the 
overall action plan in their proposal, 
including such things as the insbOlation of 
equipment and quality control systems. 
This would provide states with the 
assurance that the company has done 
everything it said it would do in the 
contract with the state. 
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CONCLUSIONS AND 
RECOMMENDATIONS 



The first objective of this project was to 
clarify the policy rationale and 
intervention model for state-financed, 
worlq^lace-based retraining programs in 
terms of their twin objectives of job 
retention and unemploymesit prevention. 
The second objective was to develop an 
evaluation system for these progranw that 
contained: 

• Business screening guidelines for 
targeting training investment and 
minimizing substitution effects. 

• A performance outcome system 
that measured the program 
outcomes necessary to achieve 
these two program objectives. 

In phase one, the project developed a 
policy rationaleand intervention model for 
retraining programs based on a review of 
two programs-California's Employmoit 
Training Panel, and Illinois' Prairie State 
20CX) Authority. Ideally, state-financed, 
workplace-based retraining progran:\s are 
designed to reduce unemployment and 
retain jobs by improving business 
performance and the competitive standing 
of the business within its industry and by 
providing marketable skills to retrained 
workers. 



Based on this intervention model, a 
successful training project is expected to 
result in the following performance 
outcome: 

Training (Behavioral) Objectives. 
Retrained workers are certified as having 
attained behavioral skill objectives that are 
designed to improve work-unit or 
company performance and enhance the 
employment opportunities of workers 
outside the company. 

Work-Unit Performance. Worker 
retraining is associated with improved 
performance of the work units that 
participated in the training project. 

Company Performance. Worker 
retraining and associated improvements in 
work-unit performance are related to 
improvements in comjany performaiKe. 

Trainee Earnings. Worker retraining 
results in stable or improved employment 
and earnings for retrained workers. 

Ba^ on this intervention model the first 
phase of the project also developed 
business screening guidelines that would 
maximize the effects of training projects on 
program objectives and minimize 
substitution problems. Seven screening 
guidelines were developed from these 
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assiimption& Three guidelines addressed 
company strategies and performance 
objectives and skill requirements. The 
remaining four guidelines addressed 
substituticn screening. These business 
screening guidelines and the four 
performance outcome objectives 
constituted the evaluation model that was 
assessed in the second phase of the project. 

The second phase of the study addressed 
the feasibility of implementing the 
evaluation model in four state-flnanced 
workplace-based retraining j ^rams. 
These programs were: (1) Cauiomia's 
Employment Training Panel, Q) nUnois' 
Prairie State 2000 Authority, (3) Missouri's 
Customized Training Program, and New 
York's Economic Development Skills 
Training Program. The fea^bility study 
was based on 24 case studies of training 
projects funded by the four state programs. 
The study also involved related research 
activities on business training and 
evaluation practices aiKi the availability of 
necessary industry and labor market 
information. TlUs second phase addressed 
four major feasibility questions. The 
findings on each feasibility question are 
summarized below. 

Business Training Design and 
Evaluation. The proposed evaluation 
system is consistent with widely 
recognized models for the design and 
evaluation of training projecte ttiat are 
intended to improvebusinras performance 
(see Hgure 4, Chapter 4). The evaluation 
system is consistent with what many 
traii^g and development professionals 
recommend as the most appropriate way 
for businesses to plan and evaluate their 
own training investments. The only major 
difference between the evaluation mwlel 



and recommended business practice is the 
model's emphasis on trainee earnings. 

Definition and Me asurement of 
Peifonnance Outcomes. The case studies 
of 24 training projects in the four state 
programs suggest that company or 
work-unit performance outcome 
indicators can be easily identified and 
measured in vaUd and reliable ways that 
are consistent with how businesses 
measure their own performance. As a 
result, such indicators would not impose 
extra costs on businesses because 
businesses normally collect the same 
information for their own internal 
purjMses, We conclude that company and 
work-unit performance are best measured 
in terms of operational objective and 
performance targets that logically linked to 
business strategies and performance goals 
(see Rgures 5 and 6, Chapter 4). The case 
studies identiBed seven major types of 
operational objectives and performance 
targets that could be linked to competitive 
strategies and business goals as well as 
financial and market performance. 

The definition and measurement o* 
trafaiing objectives varied widely in the 24 
ca^ studio However, we conclude that 
all 24 companies could have easily 
restructured their training curricula in 
terms of clearly stated and mrasurable 
learning or behavioral objectives. This 
restructuring would require additional 
effort from most companies. However, it 
is consistent with widely accepted 
standards of business training design and 
should result in more effective traii\ing 
projects. Because of validity and reliability 
problems in employee testing, we conclude 
that post-testing requirements should 
remain simple and easy to measure and 
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should address behavioral skills directly 
related to a core set of learning objectives. 

The use of trainee earnings as a major 
performance outcome indicator has 
significant problems because of the wide 
variation in company compensation 
systems. However, the employment md 
earnings of trainees is still useful to track 
and report. Data collection costs for 
business^ as well as state programs can be 
minimized by using Unemployment 
Insurance CUl) wage records. 

Business Screening Guidelines. The 

seven screening guidelines can effectively 
addx^ the most important targeting and 
substitution problems faced by state 
programs. Based on the 24 case studies, we 
found that companies can clearly state 
competitive strategy (e.g., cost, quality, 
differentiation), performance objectives, 
and the need for retraining. In most cases, 
they can cite industry benchmarks that 
they must meet to remain competitive. 
Industry information on competitive 
benchmarks is available to state programs, 
and they can use it to work with 
companies in defining meaningful 
performance objectives. With the 
exception of some small companies, most 
companies can define training objective 
and retraining requirements. State 
programs can establish simple and 
inexpensive procedures to ascertain the 
marketability of the skills acquired 
through training projects. Although the 
substitution guidelines are more difficult 
to implement, they will be effective in 
reducing some of the substitution 
problems faced by state programs. 

The seven screening guidelines can be 
effectively incorporated into a simple 
application process that would not require 



substantialextra workfbr either businesses 
or ^te program staff. The applteation 
process and the screening guidelines are 
summarized in Hgure 2 (Chapter 2). 

Im plementation of the Evaluation 
System, The proposed evaluation system 
can be administered effectively without 
imposing substantial additional costs on 
either tlK business or the program. In 
order to implement the system, ^iditional 
staff training will be required In order to 
standardize how businesses report 
performance objectives. Most ^te staff 
are sufficiently trained in identifying dear 
training objectives. 

Future business participation is difficult 
to ass^s. However, no business r^oited 
any concerns about specifying 
performance or trahiing ob^tives in 
future proposals. We recommend a 
simplified application process where each 
screening and performance outcome issue 
can be addressed effectively without a 
^gnificant increase in the requirements of 
the application procedures currentiy used 
by the four state programs participating in 
thepro^ct 

This feasibility study resulted in sbc major 
condtisions on the effects of state-financed 
training projects in the workplace. 

1. Based on interviews with 
management, trainers, and trainees, we 
condude tiia t tiaining grants generally will 
expand tiie scope of the training project, 
shorten he time it takes, or enlarge the 
number of worker involved in it In each 
of the 24 training projects examined in this 
study, the company reported that state 
assistance was key to the achievement of 
the final training product. In some 
instances, the public training subskiy was 
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the apparent reason that the training was 
actually undertaken. In most other 
instances, the businesses reported that the 
state money made it pc^ible for them to 
reach all of the workers who required 
retraining in the timeframe necessary to 
have the desired effects on company 
performance. Also, they reported that 
state funds enabled them to broaden the 
scope of training to gain the maximum 
value from the technological and 
operational improvements t»ing made in 
the company. This was accon^dished by 
morebasicand general vocational training. 

2. Training projects were most effective 
when they were tied clearly to specific 
business goals and performance objectives 
and when the training plan defined dear 
training or behavioral ob^ves. Each of 
the 24 training projects that were part of the 
study produced improvements in business 
and worker performance that were 
associated with the training. However, the 
businesses that established clear company 
and training objectives at the outset had 
highly focused training programs with 
very satisfactory results. Businesses that 
were initially less clear about their 
purposes had less satisfactory results. 
Often, the training projects appeared to go 
through periods of uncertainty and 
reevaluation and were restructured after 
objectives finally were articulated dearly. 

3. State grants have an important effect in 
overcoming management uncertainty over 
the importance of retraining workers for 
the introduction of new technologies and 
operational processes. The feasibility 
study identified a wide variety of barriers 
that prevent companies fro~> starting 
retraining projects without outside 
intervention. These include poor 
labor-management relations, bad 



experiences with prior training efforts, 
concerns that newly retrained workers will 
leave before the company is able to achieve 
a reasonable return on the training 
inv^tment, and general feelings that 
worker retraining is a luxury expense. It 
was striking that some grants were able to 
leverage significant investments in 
trainingby financially troubled businesses, 
particularly in light of the very Umited uses 
that may be made of training grants. In 
every state, training project managers in 
the businmes consistently stated that 
government assistance was instnunental 
in getting upper management to focus on 
the importance of training in bringing 
about the desired changes in company 
performance. 

4. Substitution risks are reduced 
substantially when state programs 
concentrate their grants on the direct 
instructi nal costs of laboratory and 
classroom training. Substitution risks are 
greatest when state programs pay 
companies for wages and salaries of 
trainees while they are in training. This is 
especially true for structured, on-site 
training wherein the trainees are engaged 
in productive activities that may be 
indistinguishable from their normal work 
routines after the introduction of new 
technologies or work processes. 
Consequently, it is unclear whether the 
state is subsidizing productive work or 
training. This situation differs sharply 
from formal classroom or laboratory 
training. Such instruction is always 
provided away from normal work, and it 
is usually the type of training in which 
companies have little prior experience. 

5. Company executive at the 24 projects 
that we studied, especially those operating 
small businesses, initially did not know 
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how to use training as a strategic change 
agent to improve business perfoimance. 
This problem was made more difficult 
because they had poor access to experts in 
training design and implementation. Also, 
the state agencies operating the training 
programs did not systanatically develop 
and disseminate descriptions of how 
companies may utilize training to improve 
their operations. As a coieequencev project 
managers relied on outside contractors to 
help them develop a change strategy, or 
they simply muddled through until they 
came up with a solution that seemed 
satisfactory. On a few occasions, the staff 
at the state agency provided the necessary 
technical assistance. In every stat?, the 
staff indicated tiiat they wanted to provide 
more help, but could not because of the 
workloads created by the number of 
businesses requesting training grants. 

6. State-financed, workplace-based 
retraining programs may play a major role 
in encouraging businesses to use training 
as a strategic change agent and to establish 
permanent training systems. All project 
managers reported that the retraining 
project had put training in a much more 
positive light. Consequently, upper 
management was more likely to 
incorporate training in future plans for 
making competitive improvements. As 
other businesses see the successes that are 
associated with retraining, we believe that 
state programs can beg^ to reduce the 
share of training they pay for and institute 
stricter in-kind or matching policies to 
leverage more private-sector investment. 

These major findings and conclusions 
from the feasibility study provide tiie basis 
for five recommendations. 
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1. States should adopt fhe proposed 
business screening and performance 
outcome system as 0ie basis for a more 
detailed pUm vforked out by eadt sftite in 
conformance with program statutes, 
policies, and practices. We found 0iat 
business and training mana^ uniformly 
indicated that the screening and 
evaluation model had relevance to nieir 
awn operations and ^t tfcc data ^ 
would be asked to aaatmulaU would he 
useful in their own evahations of training 
projects. In addition, we found ^t^e 
infitrmationwas importanttothe common 
fHflicy concerns of the four state programs 
participating m the study. 

2. States should recognize ^at training 
may occur in three stages: (1) basic skills 
training, (2) general occupational and 
vocational traitnng, and 0) job-specifif 
training. In these cases, states she 
establi 'i intermediate performance 
objectives for each stage. In the first stage, 
the company should be asked to show that 
workers atiiieved basic skills and, in some 
cases, introductory Udmical knowledge. 
In the second stage, the company should be 
asked to show that the workers can 
perform specific functional skiUs related 
to ultimate training objective. In the 
third stage, ihe company should be asked 
to show that training and performance 
objectives were acliiev^. However, every 
multistage training project still should be 
just^ed in terms of final business gpals 
and perfortrtance objectives. 

3. States should establish special 
assistance programs to improve the access 
of small businesses to state-financed, 
workplace-based retraining. Such 
programs should include direct 
consultation by agency staff, cooperative 
consultation by business assistance staff 
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from other state programs, cooperative 
consultation by faculty or staff at 
community oflkges or state colleges and 
universities, or a special planning and 
assistance grant that will lead to the 
development of specific business and 
training objectives for a future training 
proposal to ^agmcy. We do m>t propose 
that a special application procedure be 
instituted on behalf of small businesses. 
Instead, we propose that small businesses 
be given access to state training programs 
through assistance from Uie staff of the 
training agency, Effectively, me agency 
will have to compensate for the limited 
capacity of the small businesses. 
Although Hiis may increase significantly 
the transaction cosU incurred by some 
state agencies in Uieir efforts to readt out 
to small businesses, these costs are 
balanced by ^e value of Ike state technical 
assistance. Even if no grant is given, the 
agency has provided a useful service to the 
small business. 

4. State programs should implement the 
formative evaluation system through a 
gradual mree-^ear process: 

* Year 1. Implementation of 
business screening and 
performance outcome system with: 
a) an evaluation of the effects of 
system implementation on 
program operation including 



business partkipation, application 
screening, and contract 
management, and b) case studies of 
a represoitatlve sample of training 
projects on company performance 
and training objectives. 

• Year 2. Refinement of business 
screening and performance 
outcome system with a formal 
foUow-up stiKiy of a representative 
sample of training project on 
company performance, training 
ob^tives, and trainee earnings 
based on unemplojrment insurance 
(UD ;^ records. 

• Year 3. Implementation of the full 
follow-up system based on a 
reprsentative sample of training 
promts. Year 3 should also include 
the implementation of a system to 
track aiKi trainee earnings based on 
unemployment insurance wage 
records. 

5. Federal and state government should 
cooperate in establishing a resource center 
for disseminating information on 
state-funded training project and their 
role in improving the competitiveness of 
industry. The resounx center also should 
provide models of comprehensive training 
systems that combine basic, occupational, 
and job-specific training. 
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APPENDIX A 

STATE PROGRAMS PARTICIPATING IN 

THE STUDY 



The governors of the fifty states were 
invited by the National Governors' 
Association to participate in this study to 
develop and test an evaluation 
methodology for state-financed, 
worlq>lace-based r^raining programs. In 
order to qualify for the study, it was 
required that the state operate a program 
that had funded at least six projects that 
pn vided the following: 

1. The company receiving the 
assistance had decision-making 
authority over the content of training, 
who will provide the training, and how 
the money will be spent 

2. Formal skills training was provided 
to current employees away 6:0m the 
normal routines of work. 

3. Training of current employees was 
justified because of changing skill 
requirements resulting from the 
introduction of new technology or job 
restructuring. 

4. The purpose of training was tied 
directly to the clear and measurable 
goal of improving company 
performance (e.g., improving quality 
standards, reducing unit costs) while 
retraining workers. 



5. Hnandal support of training was 
justified based on the expectation that, 
without this support either the market 
position and long-term viability of the 
business establishment or the job 
security of the specific workers would 
be adversely affkted. 

6. Company-specific projects must 
have becni fimded withki the kst two 
years (19^ and 1988) and the training 
in these projects must be completed no 
later than 90 days before the end of the 
case study period (1989) so that 
outcomes could be tradced for at least 
90 days as part of the evaluation study. 

The Illinois Ptairie State 2O0Q Authority 
and the California Employment Training 
Panel were selected initially to partkipte 
in the study because of the long history and 
extensive experience of these agencies in 
retraining projects. They also were key 
partk:i]»nts in the early plaiuiing of the 
study. Twelve of the 24 oise studies were 
allocated to California because of the large 
size of the program and the breadth of 
funding activities; six case studies were 
allocated to Illinoiif. The remaining sbc case 
studies were c idcd equally between the 
remaining t^ . u sUi^ s. 

Four other statra responded to the request 
by the National Governors' Association; 
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New York, Missouri, Minnesota, and 
Massachusetts. Minnesota was not 
included because its res|xmse canie after 
the deadline. Massachusetts was not 
selected because the program did not meet 
all of the criteria set forth by the National 
Governors' Association. 

The four programs vary considerably in 
scope, size, project selection criteria and 
organizational structure. Although the 
missions are quite similar, each was 
established muier circumstances that were 
special to its state. 

California: Employment 
Ifeaining Panel 



The California Employment Training 
Panel was established in 1982 to: 

foster job creation, minimize 
emplojrors' unempioymmt costs, and 
meet employezs' needs for skilled 
wodcers by providing skills training 
to unemployment insurance 
claimants, recent exhaustees of un- 
employment insurance who have 
remained unemployedt, and poten- 
tially displaced workers who would 
otherwise become unemployment in- 
surance claimanis. It is the intent of 
the Legislature that all training 
funded through this ... . [pn^ram] 
result in jobs for those who success- 
fully complete the training. 

In 1989, effective on January 1, 19^, the 
California legislature amended the law to 
provide that the legislature intends that 
training funded by the panel "should make 
a substantial contribution to the long-term 
job security of the trainees." 



Previously, the legislature stated that its 
intent in establishing the Employment 
Training Panel was "to put unemployment 
insurance recipients to work by 
encouraging employers to locate and 
expand facilities in this state and training 
unemployment insurance r^ipients in 
skills needed by employers." The 1989 
amendments to the law add that the 
legislature also intends the panel "to 
prevent unemployment by increasing 
productivity through the retraining of 
existing employees." These changes follow 
actual practices of the Panel in large 
measure iimsmuch as the preponderance 
of the projects that it has funded have been 
justified on the basis of specific changes in 
com{^tive circumstances. 

The panel is composed of several 
members. Four monbers of the panel are 
appointed by the legislahire: two by the 
speaker of the assembly and two by the 
preskient pro tempore of the senate. Three 
members are appointed by the governor. 
The governor also designates the chair 
from among the members. Until January 
1, 1990, the panel hired the executive 
director, who served at its pleasiu^, and 
staff, who are subject to the State Civil 
Service Act. Beginning January 1, 19W, the 
executive director is appointed by the 
governor for a term of four years. Two 
assistant directors are appointed by the 
governor. Under prior law they served at 
the pleasure of the panel. Under current 
law, they serve at the pleasure of the 
governor. One assistant director is 
required to have experience in serving the 
needs of small businesses, with specified 
duties, and the other assistant director is 
responsible for developing and managing 
the audit and compliance program of the 
panel. 
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Prior to 1990, the panel may finance 
projects that teach people skills that will 
put them to work or that teach people^dlls 
that will keep them from losing thdr jobs. 
Grants from the fund were "only for 
training for employers or groups of 
employers who assure that those who 
successfully complete training will be 
employed." In addition, the panel must 
have been convinced that the training will 
move people into careers with long-term 
job ^curity. It would not as a matter of 
policy, train people for "deadend, 
minimum wage jobs . . . [or] . . . short term 
or high turnover jobs." Its ob^ve was to 
"help train people for good jobs that 
provide a decent living in stable or 
expanding occupations." The panel wrote 
performance-based contracts that tied 
payments to th<; successful completion of 
training and 90-day post-training 
retcntioa It reimbursed tli employer for 
all actual training costs and reasonable 
administrative expenses. However, it 
would not subsidize wages paid to trafaiees 
or pay stipends. In the event that the 
workers receiving training were union 
members, the panel required that the union 
first agree to the training before state funds 
were expended. 

The 1989 amendments provide that 
contracts may be made only fon 

training in job related basic skills, in- 
dnding literacy skills, and job related 
vocational skills that are necessary for 
participants to attain a new job or 
retain an existing job with definite 
career potential and long-term 
retraining; the identification of 
specific industries, production and 
quality control techniques, and 
regions of the state where employ- 
ment training funds would most 
benefit the state's economy and plans 



to ei^nrage trainii^ in these areas, 
including specific standards and a 
system for expedited review of 
proposals whidi meet ttie standard; 

a systm for giving an expedited review of 
proposals that are substantially similar 
with respect to employer needs, 
cuiriculunv duiatioiv and cf»ts, in order to 
encouoa^ the devdopn^t of propc»als 
that meet the needs of targeted industries 
or geographic areas; the new standards of 
accoimtability; "the research objectives of 
the panel that contribute to the 
effectiveness of the program in benefiting 
the economy of the state as a whol^" and 
"a priority list of skills that are in such short 
supply that employers are choosing to not 
locate or expand their businesses in the 
state or are importi^tg labor in response to 
these skills shortages. This list shouki 
identify those industries in which upgrade 
training is likely to encourage hiring of the 
unemployed on a backfill basis." 

Previously, the panel was required to give 
priority to employers and training for 
employers who are "expanding their 
business enterprise" in California, "to 
employers and baining for employers who 
are establishing enterprises in areas 
targeted for ecoxionnic development by the 
{California] Department of Commerce, 
and to employers and training for 
employers in industries in which there are 
critical skills shortages." In addition, 
contracts for projects involving on-the-job 
training will have to specify "the specific 
skills and competencies to be gained as a 
result of the on-the-job training component 
of the project." Contracts for new Wre 
training must require the contractor to 
provide the placement services necessary 
to ensure that trainee are placed in jobs 
for which they have been trained. 
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Finally, the panel is mandated to set 
standards of accountability for retraining 
contracts by no later than July 1, 1992, for 
all contracts (repeat contracts and larger 
contracts are sub^t to these standards 
effectivejanuary 1,1991). These standards 
will provide that all payments will not be 
considered earned until the contractor 
shows that the training has resulted in 
"measurable productivity or other 
improvements that result in a new benefit 
to the California economy. The method to 
be used for ass^ing the productivity or 
other improvements attributable to the 
training shall be specified in the contract." 
However, the amendments essentially did 
not alter the basic requirement that the 
workers receiving retraining be retained 
for at least 90 days. 

The 1989 amendments also provide that 
the Panel submit an annual plan beginning 
July 1,1990. Each plan will include: "the 
Panel's objectives with respect to the 
distribution of funds between new hire 
training and employers in which there are 
critical skills shortages." The 1989 
amendments require that the Panel give 
priority to proposals in the following 
order. 

1. New hire training and retraining for 
workers who have received notification of 
actual layoff; 

2. Retraining of eligible participants 
employed at the start of training by small 
businesses; 

3. Retraining for workers whose jobs are 
threatened by increased competition form 
outside the state; 



4. All other proposals." 

The panel is required to give special 
consideration to proposals of new 
employees of fimw locating or expanding 
in California, to new hireawi retraining for 
firms located in enterprise zones and 
economic incentive areas, and to training 
for veterans, and to training which 
supports approved apprenticeship 
programs. The panel is mandated to 
provide technical assistance to encourage 
the development of these proposals. 

Funds for the panel are derived from 
receipts from a tax imposed on employers. 
The basis for the tax is identical to 
California's unemployment insurance tax. 
Annual appropriations to the panel by the 
legislature have grown from $26 million in 
fiscal year 1983 to amounts in excess of $60 
million in fiscal years 1986 through 1989. 
However, total revenues regularly have 
exceeded their annual appropriatioiw. For 
example. In FY 1988, total revenues 
exceeded $104 million, including money 
carried forward from the preceding year 
and estimated disencumberances. 

Initial panel activities were focused on 
projects that trained unemployed workers. 
However, since its inception through June 
30, 1987, 53 percent of the projects have 
trained potentially displaced workers. 
Another 20 percent of the projects have 
combined unemployed and potentially 
displaced workers. The remaining 27 
percent of the projects trained only 
unemployed workers. 
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Illinois Fcairie State 4 
Aufliority 



The purpose of the Prairie State 2000 
Authority is to: 

establish employment training 
programs which foster J<A creation, 
reduce employer unemployment 
costs, and meet the needs of the 
economy for skilled workers by 
providing )oi>-lInked training for un- 
employment insurance claimants and 
potentially displiKed workers who 
could become such claimants. 

fhe authority was established in 1983 
under a slightly dif fierent name initially to 
operate a system of individual training 
accounts. In 1985, the mission of the 
authority was revised and two pro-ams 
were added. One program, the Individual 
Training Assistance Program, was 
established to provide finaiKial assistance 
to experienced Ul-eligible workers who 
wanted to upgrade their skills or acquire 
new ones. This program was made 
available to workers who were 
unemployed as well as those who still wete 
working. The second program, the 
Employer Training Assistance Program, 
was established "to make grants or loans to 
eligible employers for the purposes of 
provkiing training to ani^oyees in fields 
for which there are critical daonands for 
certain skills." It also provides that the 
agency provide fiiiandal assistance to an 
employen 

1) who will provide job-linked train- 
ing w^lch offers special skills for 
career advancement or which is 
preparatory for, and leads directly to, 
jobs with defhiite career potential 
and long-term job security; 



2) who is nnabte to provide suffi- 
cient funds itttenudly, ox from otter 
available MravNH^ Inch^ig fcdenii. 
State or locally administered employ- 
ment and training pr^;nuns; and 

3) (i) w]u> is ej^andii^ its businm 
mtcrpiiic In ffais St^ is locathig a 

is Intzododng more efficient ledi- 
nology into its operations which will 
mult in greater cm^tperoiipU^ce, 
is expanding into new madccts. Iff Is 
expanding exports from Illinois, and 
is ttoeby inc r easi n g tax revenues for 
State and local governments; or (ii) 
whose existing employees are 
fiureatexttd with layoff unless addi- 
tional training is made available to 
them. 

In 1989, the Ptairie State Authority 
was authorized by the Gex^ral Assembly 
to as^t employers in the preparation of a 
final needs assessment and in the design of 
a training program. The cost of the 
assessment and design may be paid fully 
by the authority axui may be made part of 
the final grant of training funds. 

The authority is governed by a 
seven-member board. Four membera are 
appointed by the governor with no more 
than two from the same politkal party. 
The other three members are ex officio: the 
state treasurer, the director of the 
Department of Commerce and 
Community AfMrs, and the director of the 
Department of Employment Security. The 
ex officio members may designate others to 
attend meetings of the board in theirplace. 
The board elects a chair from among the 
four appointed by the governor. It also 
appoints the chief executive officer, who 
serves at its pleasure, and the staff. 
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In implementing thestatute, the authority 
has used the EmployerTraining Assistance 
Program to assist employers to retrain their 
workers in new pnxxss technologic or in 
new forms of work organizatioa Pursuant 
to the statute, the authority requires that 
the employer demonstrate that it lacks the 
financial resources to conduct the 
retraining in a proper or timely fiashion. 
Hrms or establi^mente tlutt are eligible for 
assistance include those earning little or no 
profits, or those reinvesting their earnings 
in their operations. In addition, as a 
condition of assistance, the authority 
requires that the firm or establishment 
making the request provide a reasonable 
business strategy that incorporates the 
proposed retraining. Like the California 
program, if the workers receiving the 
retraining are union members, the 
authority reqmres that the union first agree 
to the retraining befo!ie the grant or loan 
request may be approved. Apprenticeship 
and training programs that are specifically 
the subject of an existing collective 
bargaining agreement are eligible for 
funding under the Employer Training 
Assistance Program. The agency will give 
grants covering 50 percent of the direct 
training costs or low-interest loans 
covering all eligible ccsts. 

Agency performance since fiscal year 
1986 shows considerable year-to-year 
consistency. In fiscal year 1986, the 
authority issut«.i 48 grants that resulted in 
retraining for nearly 4,000 workers. 
Additional grants were made to a major 
Illinois manufacturer under an 
experimental program that resulted in 
another 13,000 workers receiving 
retraining. In total, over $1 million was 
spent for employer training assistance. In 
FY 1987, the agency made 61 grant awards 
for over $937,000 and seven loans for over 



$54,000, providing training for 8,500 
employee In FY 1988, the agency made 
56 grants ($927,000) and eight loans 
($58,000) covering 9,735 workers. 

Missouzi: Hie Missouri 
Cust(MnSzedTiaii!iiig Program 

The ^fissouri Job Developn^t Fuxk! was 
established by the 83id Missouri General 
Assembly (1986) to operate two new 
programs: the New and Expanding 
Industry Training Program and the Ba^c 
Industiy Retraining Program. 

The New and Expanding Industry 
Training Program provides assistance to 
new or expanding Industrie by fuixling 
the training, retraining or uji^KKling of 
skills of potential employees. The program 
may also assist these industries by locating 
skilled employees and additional sources 
of job training funds. The program is 
funded through general state tax revenues. 

Assistance may be given to industries that 
show that their inve^ments relate directly 
to a projected increase in employment that 
will result in the need for training newly 
hired employees. It may also be given for 
retraining or upgrading the skills of 
existing employees for new jobs created by 
the investments of the new or expaiKiing 
industry. 

The Basic Industry Retraining Program, 
the subject of this study, is intended to 
provide assistance to industries in 
Missouri by supporting retraiiUng and 
upgrading of employees skills that are 
required to support new capital 
investment. Although tied to new 
investment, manufacturing investment is 
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not required to result in an incsease in 
employment in order for the indiistry to 
qualify for assistance. 

The activities that are eligible for 
reimbursement under either program 
include: the wages of instructors, 
regardless of who anploys then; training 
development costs, iiu:luding the cort of 
training instructors; training matoials and 
supplies. Including packaged training 
programs; travel directly related to the 
training program; tuition payments to 
thiid party training provkieis and to the 
indu8try;teachingandaMlstance provided 
by educational institutions in Missouri; 
on-the-job training; and lease of trrining 
equipment and space. 

The program receives oversight by the 
Missouri Job Training Join^ Legislative 
Oversight Committee. The committee is 
comprised of six members of the General 
Assembly. Three msnbcrs are appointed 
by the president pro tempore and three are 
appointed by the speaker of the house No 
more than two membeis for each house 
may be of the same political party. The 
committee reports to the General 
Assembly and the governor on all 
assistance to industries permitted inuier 
the law. 

During ascal year 1988, theBasicIndustry 
Retraining Program obligated $4,678,721 
(78 percent of the available combined 
appropriation for both the New and 
Expanding Industry Training Program 
and the Basic Industry Retraining 
Program) for training at 11 manufacturers. 
When all sources of state training 
assistance are considered, the total aniount 
of funds obligated to basic industries in the 
context of this program was $5,892,755 
during FY 1988. The Division of Job 



Development and Training and the 
Mifflouri Depaitmait of Semenlary and 
Secondary EdiKatton jointly funded nine 
of these projects in order to meet the 
retraining needs expresswi by the 
employe. Classroom trainli^ was used 
by ei^t companies; one company used 
on-the-job trainii^ak)n?;tti€balanceof the 

companies used a combination of 
classroom and on-the-job training. The 
retraining assistance involved 6,U29 
woikers. 

The average giant in the Ba^ Industry 
Retraining Program was $425338 during 
1988. This amount on average, covered 
retraining co^ of 573 peo|fe The average 
wage of each trainee was $11.72 per hour. 
The average private sector capital 
investment associated with the pro ject was 

$8,812750. 

Excluding three very large project grants, 
one each to General Motore, Chrysler 
Motors Corporation, and Ford Motor 
Company, the average pro^ cost was 
$56,124 to retrain 417 people The average 
wageofwoikeis in thesedgiht projects was 

$11.07 per hour. Associated average 
private sector capital investment was 
reported to be $5,575,437. 

NewYodcEcanoniic 
Devdopment SkOb mining 
Program 

The New York Economic Dcvdopment 
Skills Training Program was establlAed in 
1987 in response to the chaflengepiesented 
in the 1987 New Yorit Strat^ Plan for 
Economic Development that ths state 
"makeadditional and pnideit investments 

in skills development p nd woricer training 
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programs which address tiie imnffidiate 
and future needs of private sector 
employers." The strategic plan also 
enamraged new assistance to businesses to 
increase their productive capacity 
including programs for upgrading 
workers' ddlls to enhaiK% the ai^catlcm 
of advanced technologies. RnaUy, the plan 
pointed to the need for using education 
and tiainii^ to address ibe problems of 
workers who face long-term 
unemployment and economic 
disadvantage because they Uve and work 
in one of the many regions in New York 
with declining economies. 

As a consequence, the Economic 
Development Skills Training Program 
defines its mission as follows: 

To assist ittdividua! bu»i«teMM and 
indc9tric« in improving their 
pciitiveiiess by providing akiU traln- 
Ix^ to address thsi: human resource 
de'i^lapBient needs; 

To provide Job skills to enable disad- 
vantaged per$<.r»r^ dislocated 
woriccrs/ and displi»ed homemakers 
?A benefit from new jobs created 
through economic development ef- 
forts; 

To upgrade the skills of existing 
employees in order to assist New 
Yoik companies and to modernize 
and improve fiieir <^cradims and to 
meet the demands ol changing tech- 
nologies and work enviroxunents in- 
cluding assistance to employees of 
firms involved in State flnatudal as- 
sistance and industrial effectiveness 
programs; 

To provide for increased training sei^ 
vices ff» small buslneMes in ttie State, 
including new small businesses 



locating In economic development 
mul wimien Off minrattyowittd 
btudn^Mk 

The Economic Development Skills 
Training PSrogxammaymake grants to local 
training providers, including biisiness^ or 
trade associations, labor organizations, 
educational institutions, institutions of 
higher learning/ community based 
organizations^ the grant recipients or 
admlnistiative entity of a ssrvice delivery 
area, and private industry councils 
established!^ the Job Training FartnoBhip 
Act. Grants are for classroom-based 
training and on-the-^b training. 

The program is operated by a unit of the 
Department of Economic Development. 
Unlike thePrairie ^te2000 Authority and 
theEmpk>ymentTrainingFlanel, there isno 
indepoident oversight oi^anization or 
policy-setting body that governs the 
operatkm of the fnogram. However, the 
statute provides that the Commissioner of 
Economic Development convene an 
Interagency Review Committee. The 
commisstoner is chair of the committee, 
which consists of repr^entatives of the 
New York Departments of Education, 
Labor, and Social Services, the State 
University of New York, and the Job 
Training Partnership Coundi. Project 
applications not less than $25,000 are to be 
brought to the committee for its 
recommendations on actions that should 
be taken by the Commissioner of Economic 
Development. Applications for less than 
$25,000 in assistaiKe do not require the 
review of the committee. 

Projects involving on-the-job training 
proems and labor exchange or related 
functions are executed and monitored by 
the New York Department of Labor at the 
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authorization of the New York Economic 
Development Department. Qassroom- 
based training delivered by local school 
distrkts is executed and monitored by the 
Department of Education. Classroom- 
based programs delivered by community 
colleges, agricultural and technical 
colleges, or public degree-granting 
institutions of higher education are 
executed and monitored by the State 
University of New York or the City 
University ofNewYorlgas appropriate. In 
addition, the Commissioner of Economic 
Development has created regional 
networks in each of the state's 10 economic 
development regions. Involving local 
experts in thedelivery of theSkillsTraining 
Program, to bring about greater 
coordination of resource and increased 
responsiveness to regional economic 
development needs and initiatives. 

The statute establishing the skills training 
program requires that one-half of its funds 
must be used to provide assistance to small 
businesses with fewer than 100 employees. 
One-half of the funds must be used to 
provide skills training to targeted 
individuals (disadvantaged people, 
dislocated workers, displaced 
homemakera). Particular emphasis is 
given to assist minority- and women- 
owned firms as well as to train and 
upgrade the skills of women and 
minorities for improved employment 
opportunities. Program staff are required 
to consult the appropriate labor groups 



wherever there is a collective baigaini .^; 
agreement in effect with an employee 
partkipating in the progiam. Inaddltloxv 
notification is given to the appropriate 
local Private Industry Couxidl for any 
project funded in its area. 

The skills training program operates with 
anannualappropriation from general state 
tax revenues of $4.4 million. The 
Dq»rtment of Economic Dev^pmsit is 
mandated by law to work coopoatively 
with other agencies by sharing 
responsibility for training costs. Training 
c(»ts include the direct cost of instruction 
and may include an on-the-job training 
wage reimbursement not exceeding 50 
percent of the wage costs for a maximum 
of 12 weeks. Each dollar expended by the 
program for training must be matched by 
at least one dollar from other federal^ state, 
local, or private resources. The full amount 
of assistaxKe to a business may be paid to 
the applicant only if it has adiieved its 
anticipated outcomes. Thoe outcomes are 
defined in the project application and may 
specify, as appropriate, "job placement 
rates, number of jobs opened to targeted 
populations as a result of skills upgrading 
activities, [and] promotions or wage 
increases for individuals partidpating in 
skills upgrading programs." IiKiividual 
businesses receiving the training also must 
provide a commitment to hire from among 
individuals who have successfully 
completed ti'aining. 
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SAMPLING DESIGN AND METHODOLOGY 

FOR CASE STUDIES 



Case Sdection Criteria 

Following the selection of the four states, 
we reviewed case descriptions confined 
in annual reports for Illinois and 
Califbmia-the two programs that had 
funded a large enough number of cases to 
pomit the application of some sampling 
design. These cases could be organized 
into nine types: five types were specific to 
manufacturing production activities, and 
four types were specific to service 
functions, either in service Industries or as 
service support activities within a 
manufacturing hisiness. In addition, each 
of these nine types could be separated by 
size into large and small. The nine 
categories of cases were as follows 
(examples of each category of cases are 
noted parenthetically): 

ManujFacturing Industry/Activities 

1. New Process Technologies, Work 
Procedures, Supervisory Practices 
(Simpson; Lawrence Box & Basket; KLM; 
Areata Graphics; Pirelli-Armstrong; 
LA-NTMA; Solar Turbines; AT&T; 
NUMMI; GM-Wentzville; Dunlop). 

2. Quality Conti-*l Procedures 
(Manth-Brownell; Harmon; NUMMI; 



Pirelli-Armstrong; Lyphomed; Ingersoll 
Milling Machine). 

3. Job Rotation, ^ob Sharing, Job 
Enlargement, Cross-Training 
(Dresser-Rand; Northwestern Steel; 
AT&T; Manth-Brownell; GM-WentzvUle; 
Solar Turbines; Simpson; Areata 
Graphics). 

4. Computer Aided D^ign ami Drafting 
(Unicadd; Metcraft; Solar Turbines). 

5. ProductionandMaterials Management 
(Rohr Industries; KLM). 

Service Industxy/Administration 
Support and Services 

6. OfBce Support Automation (Glendale 
Community College, Pirelli-Armstrong). 

7. Merging /Upgrading Management 
Information Systems (Care Enterprises). 

8. New Ord^lng, Inventory, Processing 
and Sales Information Systems (Thrifty; 
Rohr Industries). 

9. Management Sales,ServiceTraining (or 
New Products and Services (United 
Savings Bank). 
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R>llowing devebpment of the categories 
of cases, each state participating in the 
study was asked to provide suggestions of 
those cases that it considered to be above 
average. Our purpose in selecting the 
better cases was ba^ on the assumption 
that in order for an evaluation system to be 
considered toible and to be accepted by 
the state, it must be applied successfully to 
at least those cases that the state considered 
to be a good reflection of the value of the 
program. This also was in retxignition of 
the fact that in addition to checking the 
feasibility of our proposed evaluation 
system, an important purpose for this 
study was to give illustrations of the types 
of projects that these four states were 
financing in order to help others 
understand the work of these agencies 
better. 

After we received the suggestions of the 
states, the companies were contacted by 
phone by one of the researchers 
(occasionally representatives from the 
state agencies also made phone contact) to 
inform them that their respective states had 
recommended them for the shidy. This 
was followed by a letter explaining the 
project and containing sample questions, 
and a schedule was sent to the company 
contacts identified by the stat». The letters 
and accompanying materials were 
standardized by size of company. In a few 
Instances, the grant recipient also was 
visited in order to explain tt\e study and to 
set the schedule for die site visit. 

Several companies refused to participate 
in the study. Their reasons varied: 

• One California food processor 
refused because it was a closely 
held company that was adverse to 
any publicity about its operations. 



• One California bank refused to 
participate because all 
management and training 
personnel associated with the 
training pro^ had left the bank. 

• Two CaUfbmia retailers refused to 
participate and gave no reason. 

• Two Illinois manufacturers refused 
to participate because of 
scheduling conflicts. 

• One Illinois company did not 
participate Ijecause of lankruptcy . 

• One New York manufacturer 
refused to participate and gave no 
r^son. 

• One Missouri manufacturer did 
not participate because of 
scheduling conflicts. 

In addition to these companies refusing to 
participate, one Illinois company received 
a site vi^t but later was dropped from the 
study. The reason it was dropped was that 
a needed second site visit could not be 
scheduled and because of ad verse business 
conditions being experienced by the 
company. 



Site Visit Procedux^ 



The site visit generally was spread across 
two days, not necessarily consecutive days. 
Two comj^nies refused to permit more 
than one visit; four companies-two with 
fewer than 15 employees-required only 
one site visit. Each visit began with an 
orientation interview and company tour 
with the principal contact at the comf.«ny, 
usually the training manager (in the case of 
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small companies, this was the company 
owner). Often this interview and tour 
required two hours. Following the 
orientation interview, interviews were 
requested of upper management (in the 
cases that it was not the company contact), 
supervisory personnel, workers receiving 
the training, the training provider, and 
union officials (if appropriate). 

Although all companies participath>g in 
the study were ccK>perative, we were not 
succ^ful in every instance in conducting 
all the necessary interviews. Often we 
were unable to interview the training 
provider because it no longer was 
providing services to the company aiKi 
could not be brought in for the study. 
Upper management also was not 
interviewed on occasioa The usual reason 
was that the managers that were present at 
the time the training occurred either had 
left the company or were transferred to 
another facility. Union officials were not 
intei-viewed in those instances where the 
participation of the union was incidental to 
the training project (i.e., the approval of the 
union was received for purposes of the 
training grant but it took no active role in 
any aspect of the training project). 



Following the site visits, each conqpany 
was contacted by letter or by phone for 
additional information. It was requested 
to provide specific information abcmt pet^ 
formance indicators and changes in pei^ 
fbrmance, the competitive strat^ of the 
firm^ the organizational structure of the 
company, and fads that were undear or 
appeared inconsistent with each other. 

Case write-ups were based on the 
information collected in the interviews or 
provided by the company and from the 
case files of the state training ageiKles. 
Each company received a copy of the 
write-up for the company and was asked 
to review it for Actual errors and to 
identify any information that it considered 
proprietary. Such proprietary information 
would be adjusted so that it would not 
reveal trade secret information and still 
make the point about the change in 
performance indicators associated with the 
trairting intervention. Followix^ company 
approval of the case write-up, all cases 
were submitted to the respective states for 
their review and approval. 
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IMPACT EVALUATION OF RETRAINING 

PROGRAMS 



Impact evaluation addiesses the most 
fundamental evaluation question in public 
programs. It addresses Uie validity of the 
intervention model. It assesses whether 
desired outcomes are different torn what 
wouW be expected without the picgiam 
and whether these outcomes can be 
attributed to program interventions. In the 
case of state-ananced, workplace-based 
retraining programs, impact evaluation 
would address whether worker retraining 
Increased job retention and reduced 
unemployment. 

Despite the importance of impact 
evaluation, this type of evahiation is very 
difficult to do for most ongoing 
femployn.ent and t.-aining and economic 
development programs. Net impact 
evaluations are rare in state economic 
development pn^ms. The history of net 
impact evaluation in employment and 
training programs has been marked by a 
continuing debate on the relativestrcngths 
and weaknesses of experimental versus 

quasi-experimental research designs. This 
appendix summarizes pro^ findings on 
the impact evaluation of retraining 
programs using experimental and 
quasi-experimental deigns. 



Experimental Designs 

Experimentaldesigns require thenmdom 
sdection of pn^iam partteipants from a 
pool of eligible potential participants. 
Random assignment has been 
controverelal in anployment and training 
programs but has been used extensively in 
the evaluation of JTPA programs and 
welfare-for-work demonstrations. The 
major disadvantage in using eqjerimental 

designsforretiBiningprogramsistiiattiiey 
would require tiiat programs deny or 

signifkantiy delay services to a random set 
of businesses. 

Random selection would be extremely 
difficult to implement in state-financed, 
workplace-based retraining programs. 
These are relatively new •jrograms that 
continually confirontsensitivededsions on 
which businesses should receive state 
retraining funds. MoiU state programs are 
in the process of defining new targeting 
and screening guidelines in tiw face of a 
growing businras demand for training 
funds. Establishing impact evaluation 
systems ba jed on random selection would 
probably be detrimental to the future 
development of tiiese programs. Random 
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selection may be an option in the future as 
programs develop clear targeting and 
screening guidelines and progxan^ receive 
a stable level of business funding 
applications that exceed the money 
available for retraining assistance. 



Qiiasi-Experimental Designs 

Quasi-experimental designs develop 
matched control groups for comparison 
with program participants. Although 
random assignix»nt provides a stronger 
evaluation design with more internal 
validity, quasi-experimental designs are 
sometimes more practical because they do 
not require denial or delay of services and 
can be conducted without major intrusion 
into a program. Although comparison 
group designs have been used extensively 
in the evaluation of emplo3anent and 
training programs (e.g., Bamow, 1986; 
State of Washington, 1987), they have yet 
to be developed for economic 
development programs, especially 
programs that retrain employed workers. 
One exception has been a series of studies 
for California's Employment Training 
Panel using non-completors as a 
comparison group for evaluating worker 
earnings gains (Training Research 
Corporation, 1987). Michigan also is 
exploring comparison group designs for 
evaluating business performance 
improvement using businesses that had 
applied to its retraining programs after all 
money had been obligated for the fiscal 
year. 

Quasi-experimental impact designs face 
four maior problems in addressing 
retraining programs. The first major 
problem is developing comparison groups 
of businesses and workers. There 



currently is not a comprehensive data 
source in states for developing a 
comparison group of businesses that is 
similar in all key characteristics to 
businesses partidpatii^ in state programs. 
Employn^t Kcurity ou^x^s fites (Le., 
E5202) provide a comprehensive data 
source for matching by industry and size 
but do not contain other essential 
information that can msure the 
comparability of groups. Son^ states may 
be able to build comparison groups from 
businesses whose applications are being 
held over to the following year because of 
lack of iiinds. However, this will not be 
practical in all states. In addition, 
bu^esses may receive funding as soon as 
six months later in the begimiing of the next 
fiscal year. This may be too ^ort of a delay 
period to assess what businesses wouhi 
have done on their own without state 
fiuMling assistance. Third, business^ that 
do not receive state funds for retraining 
workers n^ay not be willing to provide 
reliable information on training e^ris and 
com{»ny employment and performance. 
Busing surveys generally have a much 
lower response rate than worker surveys 
that track employment and earnings. 
Fourth, previous research on worker 
retraining has yet to identify the major 
btisiness and worker characteristics that 
must be considered iu ^timating the 
impact of public programs. Impact studies 
of employment and training programs can 
rely on a long history of labor market 
research to identify critical causal factors 
that explain employment and earnings. 
This is not true for impact evaluations of 
programs that retrain employed workers. 

In summary, we conclude that impact 
evaluation systems based on either 
experimental or quasi-experimental 
deigns are not feasible at this time for 
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Ftate-financed, workplace-based 
retraining programs. States should 
concentrate most o£ their efforts on 
developing a process monitoring and 
performance outcome assessment system. 



However, because of th'^ in^rtance of 
impact evaluation, states should ^poisor 
special evaluation studies that can provide 
the foundation for a future impact 
evaluation system for retrainingprc^rams. 
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